Closed agstephens closed 3 years ago
I have implemented a script:
https://github.com/glamod/glamod-cds-forms/tree/master/constraints/add-days-to-counts.py
This writes counts
files that include all the days for each month.
Next, need to re-run the minimiser script.
That approach failed - was running python process for days trying to minimise constraints - which was greater than 4Gb in memory usage.
Second approach:
Prototyping in:
/usr/local/glamod-cds-forms/constraints/expand-constraints-by-day-hour.py
Latest actions, all combined here for now:
(surface-land|surface-marine)_(monthly|daily|sub-daily)_<start_date>_<end_date>_(global|subset)_csv-obs_<hash8>.csv
(surface-land|surface-marine)_(monthly|daily|sub-daily)_<start_date>_<end_date>_(global|subset)_data-policy_<hash8>.txt
- where: <hash8> is eight random characters.
SELECT * LIMIT 1
instead of SELECT COUNT(*)
- should be quicker v1/
web service - in a static file URL>>> y = 1700
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: offset must be a timedelta representing a whole number of minutes, not datetime.timedelta(-1, 86325).
>>> y=1850
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
datetime.datetime(1850, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1850, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(1800, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1800, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(1700, 1, 1, 0), '%Y-%m-%d %H')
datetime.datetime(1700, 1, 1, 0, 0)
>>> datetime.datetime.strptime('{}-{}-{} {}'.format(y, 1, 1, 0), '%Y-%m-%d %H').astimezone(UTC)
datetime.datetime(1850, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
This has been improved/fixed in commit: https://github.com/glamod/glamod-cds-forms/commit/dd849253826e7bd55dd9edf938b8f5166175f0b4
First approach: