Open charliepascoe opened 3 years ago
Here is the directory structure for CCMI-1
data/CCMI-1/output/<institution_id>/<model_id>/<experiment_id>/<frequency>/<modeling_realm>/<mipTable>/<ensemble-id>/<version>/<variable>
e.g. data/CCMI-1/output/MOHC/HadGEM3-ES/refC1/mon/atmos/monthly/r1i1p1/v1/zmo3
The only thing I would change about the directory structure for CCMI-2022 would be in the way we implement the version directory. I would recommend that we use the date of the upload (yyyymmdd) as the version identifier rather than an incremental integer. In this way the CCMI-2022 directory structure would be more similar to the conventions used by CMIP6.
I am quite happy with the suggestion to move from a version number to a date. I noticed the CMIP6 DRS specifies the
I just realized my previous comment was missing an important word. Where I was commenting about what I felt was a 'redundant' directory level, I was referring to the
Maybe it wasn't me... Seems when I put the word 'frequency' in angle brackets it disappears. Maybe <_frequency_> works better?
I agree that the <frequency>
directory seems redundant in the directory tree (given that we also have the <mipTable>
directory. However, given that we want to be able to make the CCMI-2022 directory structure compatible with the ESGF CMIP6 data I think we need to keep it.
Maybe it wasn't me... Seems when I put the word 'frequency' in angle brackets it disappears. Maybe <_frequency_> works better?
Yes it is weird I'm using the "insert code" feature from the menu above the comment box to make the content my angle bracket text appear.
With respect to the directory structure of the archive, we have a little more time to work that out.
Aside from the version, the filenames will have sufficient information to be able to place them in the directory tree. I use the unix find
command to select the files that people upload so it really doesn't matter what structure they arrange them in. They can either upload a flat list of files or files in a directory tree.
My suggestion for structure of the upload area is that the users make a ccmi-2022 and then version directories beneath that for each batch of files they upload.
Sounds good. I'll add in the directions of how to submit data that reflect your comments. So when a user creates an account and logs on to upload data would they be in their own subdirectory or a shared space? To put the question another way, do they need to create an initial account along the lines of <source_id>
?
When a user logs in to upload data they would be in their own subdirectory on our arrivals machine. I then look in their personal directory on the arrivals machine to find their ccmi-2022 data uploads.
I am trying a test upload to CEDA now, using the web interface on arrivals.ceda.ac.uk. I have created a new delivery as the first step, which takes me to the page where I need to assign a name. When I give the delivery the name 'ccmi-2022' I can see that the name becomes part of the directory structure, and if I am using ftp (or rsync, I assume?), users can then create a version subdirectory (vYYYYMMDD) and deposit data under that. I have done that and uploaded a single file by ftp. But when I come back to the web page and go to 'Review submission' I am told the delivery was not submitted because of the the missing metadata.yaml file. If the CCMI data submission requires a yaml file, can we provide groups with a template?
One other question I have - Under 'Other upload methods' for ftp or rsync the system generates a new password. Does this password only affect the arrivals machine for ftp or rsync? The next time I go to log on to the arrivals web interface, will I need to use the new password? And is there an option to use my original password for the data transfer or must users always click on 'Generate new password' and use the system generated one?
I've just had a look in your dplummer directory in our arrivals/users area and I can see the toz_monthly_CMAM_refC1_r1i1p1_196001-201012.nc file that you uploaded today so it looks like that part worked.
The metadata.yaml file is used to collect information for the catalogue records. As far as I am aware, the .yaml file is not a requirement for uploading data but we can use yaml files as a method for the ccmi-2022 modelling groups to provide information for their catalogue records.
I haven't heard of the other upload methods generating new passwords, I'll follow that up with my colleagues in the morning.
Through the web interface I can see the file is on the arrivals machine, but then the website asks me to review and confirm submission before submitting delivery and that is where the process is stopped by the requirement for the yaml file. Maybe I am making this too complicated by going through the web interface? Is it okay if users just ftp or rsync to arrivals.ceda.ac.uk and create the required directories (CCMI-2022/vYYYYMMDD) and deposit data there?
I can also add that while I can still log on to my CEDA account using my original password, this does not work for the ftp connection to arrivals.ceda. I now need to use the password that was generated by the upload page when I clicked on "Other upload methods > ftp".
I just double-checked the CMIP6 DRS document and the directory structure is given as:
<mip_era>/<activity_id>/<institution_id>/<source_id>/<experiment_id>/<member_id>/<table_id>/<variable_id>/<grid_label>/<version>
This would then be the exact same structure used here?
Yes, we'd use the same directory structure.
Martin noticed that the ccmi-2022 CV has set cmip6 as the mip_era, he has suggested we use post-cmip6 instead https://github.com/cedadev/ccmi-2022/issues/8
comparing with our cmip6 data the ccmi-2022 dir structure will be
ccmi-2022/data/<mip_era>/<activity_id>/<institution_id>/<source_id>/<experiment_id>/<member_id>/<table_id>/<variable_id>/<grid_label>/<version>
The ccmi-2022 CV has mip_era set to cmip6. Our archive structure has mip_era set to post-cmip6. Is it a problem for downstream software if they are not the same? https://github.com/cedadev/ccmi-2022/issues/8
Agree a directory structure for CCMI-2022