Need a synthetic CFA file for unit tests

bnlawrence / cfs

Rebooting cfstore, leaner and meaner

0 stars 1 forks source link

Need a synthetic CFA file for unit tests #29

Open bnlawrence opened 1 month ago

davidhassell commented 1 month ago

Hi Bryan - is the extra trailing dimension (of size 2) in the fragment file names needed for this file? I thinking not, which will simplify things.

davidhassell commented 1 month ago

Sample attached, based on just the first 2 fields in CANARI_999_cn134_atmos.nc. Does that look like the sort of thing? Can do this for any of the fields in the full file. sample_cfa_test_file.cfa.gz

bnlawrence commented 1 month ago

That's great. Do you think it would be possible to change the locations of some of the data so we have a combination of data which is unavailable, some which is tape, and some of which is on tape and on disk, so we can exercise the tools for that sort of thing?

bnlawrence commented 1 month ago

Hi Bryan - is the extra trailing dimension (of size 2) in the fragment file names needed for this file? I thinking not, which will simplify things.

To be honest I am not sure what that was doing anyway ...

bnlawrence commented 3 weeks ago

Following up our discussion today.

We want to be able to support, and test, the following use cases:

CFA file has all fragments in the same location, which has a substitution variable which we can change
Update half of those fragments to a different location
Update a different half to include a second fragment at a different location (We may then have between 1 and 3 fragments locations per fragment, and three possible substitution values.)

We talked about a new method similar to get_filenames, but instead of returning a set, would return an array of fragments and how they map onto the aggregated variable space (in order, with the multiple fragments).

davidhassell commented 2 weeks ago

(I am looking at this)