cross-compatibility - Githubissues

quiltdata / quilt

Quilt is a data mesh for connecting people with actionable data

https://quiltdata.com

Apache License 2.0

1.32k stars 91 forks source link

Open drernie opened 1 month ago

drernie commented 1 month ago

Adapt test_spec to:

Create packages of different sizes: 0 MiB | 5 MiB (S, M) | 8 MiB (S, M, V) | 8 MiB + 1
Using different checksum types: monolithic | chunked
Compare with reference packages from the Catalog

drernie commented 1 month ago

Requires access to: quilt+s3://udp-spec#package=spec/parquet@f3085fd3355ca8ff22d4caac56abb4dc783258dc3910670f96d571bea664099e

drernie commented 1 month ago

Tests
B: Browse	work with manifests (with \| without) chunksums (local \| s3)
V: Verify	work with manifests (with \| without) chunksums (local \| s3)
U: Uploaded	package creation with upload
E: Existing	package creation from existing data (valid etag \| sha256 \| other)
O: Optimized	upload optimisation (w/KMS \| wo/KMS)
	object not re-uploaded if quilt3, S3 objects have checksums
	e.g. check LastModified

drernie commented 1 month ago

Namespace created packages by quilt3 version.

drernie commented 1 month ago

Proposed test matrix

drernie commented 1 month ago

Test files (on s3://udp-spec) spec-setup.csv