search

manatee-project / manatee

Apache License 2.0
15 stars 3 forks source link

[Refactoring] API should not use its local file system to handle requests #3

Closed dayeol closed 1 week ago

dayeol commented 1 month ago

Currently, the DCR API is using its own file system to create a build context (a tar file) for a Kaniko job.

https://github.com/manatee-project/manatee/blob/757406e28e9af4796683e4aa2ddc9a713e5bfc44/app/dcr_api/biz/service/kaniko_service.go#L52-L107

This is not ideal because of a few reasons: (1) Performance - File I/O is extremely slow, so it will likely increase the overall API latency by an order of magnitude. (2) Statelessness - API needs to remain stateless, where using file system obviously make it hard (3) Security - using file system makes requests essentially share the same system resources, leading potential security risks.

Change it not to use the file system. Instead, we can use bytes.Buffer and tar.NewWriter to do everything in memory.

reference: https://go.dev/src/archive/tar/example_test.go

dayeol commented 2 weeks ago

Hopefully this should be fixed when we fix #23