Update "constraints" package so that it sets constraints for day of month based on the year and month.

[agstephens]

First approach:

1. Take the counts file
2. Expand/update the counts by adding a line per day (for each year and month)

[agstephens]

I have implemented a script:

https://github.com/glamod/glamod-cds-forms/tree/master/constraints/add-days-to-counts.py

This writes counts files that include all the days for each month.

Next, need to re-run the minimiser script.

[agstephens]

That approach failed - was running python process for days trying to minimise constraints - which was greater than 4Gb in memory usage.

[agstephens]

Second approach:

1. Parse the final constraints file.
2. For each dictionary in the constraints:
   • generate a list of all year/month combinations
   • group them by the number of days in the month
3. For each group member:
   • create a new entry that also includes:
   • day: [01...]
   • hour: [01...23]
     4. Write them all to a new JSON file.

[agstephens]

Prototyping in:

/usr/local/glamod-cds-forms/constraints/expand-constraints-by-day-hour.py

[agstephens]

Latest actions, all combined here for now:

• [x] exclude hour from constraints
• [x] fix the invalid days (e.g. 30th February - where doesn't exist)
• [x] regenerate the constraints.json file
• [x] regenerate the marine constraints file
• [x] add more meaningful names in zip file and contents:
  • (surface-land|surface-marine)_(monthly|daily|sub-daily)_<start_date>_<end_date>_(global|subset)_csv-obs_<hash8>.csv
  • (surface-land|surface-marine)_(monthly|daily|sub-daily)_<start_date>_<end_date>_(global|subset)_data-policy_<hash8>.txt - where: <hash8> is eight random characters.
• [x] sort (cluster) the tables
• [x] if there is no data in a response (e.g. a bad year):
  • return an empty table
  • return an empty data policy file
• [x] modify code for checking whether a given variable exists for a time, frequency, domain combination:
  • use: SELECT * LIMIT 1 instead of SELECT COUNT(*) - should be quicker
• [x] serve each constraints file from our v1/ web service - in a static file URL
• [x] check requests for years out of range will work (e.g. 1033-1066) -
  • This needs a fix because timezones are not working (see below)
  • Python can't handle timezone UTC pre-1800. So need to change format of time into SQL, in sql_mngr.py

>>> y = 1700
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: offset must be a timedelta representing a whole number of minutes, not datetime.timedelta(-1, 86325).
>>> y=1850
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
datetime.datetime(1850, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1850, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(1800, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1800, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(1700, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1700, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
datetime.datetime(1850, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)

[agstephens]

This has been improved/fixed in commit: https://github.com/glamod/glamod-cds-forms/commit/dd849253826e7bd55dd9edf938b8f5166175f0b4