No direct way to only load the artifact metadata without the BLOB, meaning a lot of data transfer we don't usually care about
No way to use an external storage tool, such as an S3, to store the BLOBs. BLOBs are not really meant to be stored in databases as you can't query them and they just bloat the database leading to worse performance.
Suggested
Add new model database.ArtifactData
Change Data field in database.Artifact to be a pointer to database.ArtifactData
Only load the Data when needed, such as in the GET /build/{buildId}/artifact/{artifactId} endpoint
Add migration to move the data to the new table artifact_data
Future possibilities
Add S3 support. Adding migrations should be easy enough. We could change the ArtifactData.Data field contain reference to the artifact ID in the S3 storage.
In our internal cluster we have built-in S3 support via Rook Ceph, so we could take good advantage of this.
Still needs the database support as fallback for those without S3 storage available.
Two main issues:
No direct way to only load the artifact metadata without the BLOB, meaning a lot of data transfer we don't usually care about
No way to use an external storage tool, such as an S3, to store the BLOBs. BLOBs are not really meant to be stored in databases as you can't query them and they just bloat the database leading to worse performance.
Suggested
database.ArtifactDataDatafield indatabase.Artifactto be a pointer todatabase.ArtifactDataDatawhen needed, such as in theGET /build/{buildId}/artifact/{artifactId}endpointartifact_dataFuture possibilities
Add S3 support. Adding migrations should be easy enough. We could change the
ArtifactData.Datafield contain reference to the artifact ID in the S3 storage.