search

cioos-siooc / cioos-pacific-erddap

CIOOS Pacific ERDDAP main repository used to maintain, develop and integrate ERDDAP datasets.
https://data.cioospacific.ca/erddap
2 stars 0 forks source link

Failed to convert UBC files #11

Open JessyBarrette opened 1 year ago

JessyBarrette commented 1 year ago

The following files failed to be converted. Related to HakaiInstitute/cioos-pacific-erddap-public#19 www.waterproperties.ca/osd_data_archive/UBC/7816/78160009.UBC Failed to convert file

Traceback (most recent call last):
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 251, in get_date
    date_obj = datetime.strptime(date_string[4:], "%Y/%m/%d %H:%M:%S.%f")
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '1978/10/19 23:17:44' does not match format '%Y/%m/%d %H:%M:%S.%f'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cioos/cioos-siooc_data_transform/projects/ios_data_transform/ios_data_transform_script.py", line 108, in unpack_file_convert_threads
    convert_files_threads(*inputs)
  File "/home/cioos/cioos-siooc_data_transform/projects/ios_data_transform/ios_data_transform_script.py", line 128, in convert_files_threads
    if fdata.import_data():
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 1247, in import_data
    self.start_dateobj, self.start_date = self.get_date(opt="start")
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 254, in get_date
    date_obj = datetime.strptime(date_string[4:], "%Y/%m/%d")
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains:  23:17:44

www.waterproperties.ca/osd_data_archive/UBC/6114/61140001.UBC Failed to convert file

ERROR - 2023-03-10 21:07:14,054 - ForkPoolWorker-1:24188 [code= "/home/cioos/cioos-siooc_data_transform/projects/ios_data_transform/ios_data_transform_script.py", line 110; file="/data/ios_raw_files_temp/www.waterproperties.ca/osd_data_archive/UBC/6114/61140001.UBC"] Failed to convert file
Traceback (most recent call last):
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 251, in get_date
    date_obj = datetime.strptime(date_string[4:], "%Y/%m/%d %H:%M:%S.%f")
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains:            !DERIVED FROM INVALID TIME

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cioos/cioos-siooc_data_transform/projects/ios_data_transform/ios_data_transform_script.py", line 108, in unpack_file_convert_threads
    convert_files_threads(*inputs)
  File "/home/cioos/cioos-siooc_data_transform/projects/ios_data_transform/ios_data_transform_script.py", line 128, in convert_files_threads
    if fdata.import_data():
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 1247, in import_data
    self.start_dateobj, self.start_date = self.get_date(opt="start")
  File "/home/cioos/cioos-siooc_data_transform/cioos_data_transform/cioos_data_transform/IosObsFile.py", line 254, in get_date
    date_obj = datetime.strptime(date_string[4:], "%Y/%m/%d")
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/home/cioos/anaconda3/envs/cioos_data_transform310/lib/python3.10/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains:  09:01:00.000           !DERIVED FROM INVALID TIME
JessyBarrette commented 1 year ago

This is now resolved

JessyBarrette commented 1 year ago

The issue doesn't seem to be resolved on the development server itself Sentry issue: IOS-DATA-TRANSFORM-W