search

oceansites / dmt

Activities of the OceanSITES Data Management Team
http://www.oceansites.org/data
6 stars 1 forks source link

GDACs create site report by crawling THREDDS catalog #12

Open dpsnowden opened 8 years ago

dpsnowden commented 8 years ago

Determine a SITE by SITE report that we can use to see how current submissions are. Possibly from the index files. Jing/Thierry

dpsnowden commented 8 years ago

@MBARIMike can you share the thredds_crawler code that you showed me? I think we'd like to crawl the catalog and generate stats for each file and/or site. The crawler plus compliance-checker could be useful.

MBARIMike commented 8 years ago

I can throw something together and put it somewhere here. Should we make a directory? Something like "scripts", "python", or "utilities"?

dpsnowden commented 8 years ago

Don't know what makes the most sense. If we are successful in collecting several utilities we should probably have separate repositories if they are truly separate pieces. On the other hand, there's a learning curve.

I was thinking that this repo was primarily ofr tracking high level actions and not all the issues associated with an individual script/application that we want to jointly develop.

Maybe a collection of scripts here is fine. But something specific like what I mentioned in #27 might deserve it's own repo.

MBARIMike commented 8 years ago

I'll start a collection of scripts in a utilities directory here. If it grows too big we can graduate it to another repo. There's a balance to achieve between a proliferation of repos as well...

MBARIMike commented 8 years ago

The issue https://github.com/asascience-open/thredds_crawler/issues/16 needs to be addressed before my site_report.py script will work well for others.

MBARIMike commented 6 years ago

During the Kiel meeting the compliance_report.py script was updated to work in an Anaconda Python 3.6 environment using the current IOOS compliance-checker API. See https://github.com/oceansites/software/issues/4 for the details.

It would be nice to have the script run faster. I'll probably open another software issue to track that effort.