sot / cheta

Cheta Telemetry Archive
https://sot.github.io/eng_archive
BSD 3-Clause "New" or "Revised" License
3 stars 4 forks source link

Add overwrite=True to cheta sync index table write statement #248

Closed taldcroft closed 1 year ago

taldcroft commented 1 year ago

Description

Daily cheta cron processing failed when updating the cheta sync archive. This is due to a missing overwrite=True arg in the astropy Table write() method. This was not seen in testing because the test only does one "update" into a fresh directory. This could be improved...

2023-03-29 04:49:05,151 Writing 1 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
Traceback (most recent call last):
  File "/proj/sot/ska3/flight/bin/cheta_update_server_sync", line 33, in <module>
    sys.exit(load_entry_point('cheta==4.58.0', 'console_scripts', 'cheta_update_server_sync')())
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 135, in main
    update_sync_repo(opt, logger, content)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 187, in update_sync_repo
    index_tbl = update_index_file(index_file, opt, logger)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/cheta/update_server_sync.py", line 335, in update_index_file
    index_tbl.write(index_file, format="ascii.ecsv")
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/table/connect.py", line 130, in __call__
    self.registry.write(instance, *args, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/registry/core.py", line 385, in write
    return writer(data, *args, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/ascii/connect.py", line 28, in io_write
    return write(table, filename, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.10/site-packages/astropy/io/ascii/ui.py", line 940, in write
    raise OSError(NOT_OVERWRITING_MSG.format(output))
OSError: File /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv already exists. If you mean to replace it then use the argument "overwrite=True".

Interface impacts

None

Testing

Unit tests

Independent check of unit tests by Jean

Functional tests

On HEAD

ska3-kady$ cheta_update_server_sync  --sync-root ${SKA}/data/eng_archive --content=acis2eng
2023-03-29 06:15:20,234 Writing 1 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
2023-03-29 06:15:21,881 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/full.pkl.gz with 2603 rows of data and 106 msids
2023-03-29 06:15:23,395 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/5min.pkl.gz with 261 rows of data and 101 msids
2023-03-29 06:15:25,958 Writing /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-03-28T0245z/daily.pkl.gz with 0 rows of data and 101 msids
2023-03-29 06:15:25,963 Removing sync directory /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/2023-01-29T0334z
2023-03-29 06:15:25,981 Writing 60 row(s) to index file /proj/sot/ska3/flight/data/eng_archive/sync/acis2eng/index.ecsv
2023-03-29 06:15:26,021 Writing contents pickle /proj/sot/ska3/flight/data/eng_archive/sync/msid_contents.pkl.gz

On laptop

(ska3) ➜  ~ cheta_sync --content acis2eng
2023-03-29 06:17:17,538 Running cheta_update_client_archive version 4.58.0
2023-03-29 06:17:17,539   /Users/aldcroft/miniconda3/envs/ska3/lib/python3.10/site-packages/cheta/update_client_archive.py
2023-03-29 06:17:17,539 
2023-03-29 06:17:17,539 Updating client archive at /Users/aldcroft/ska/data/eng_archive
2023-03-29 06:17:17,801 Reading index file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/index.ecsv
2023-03-29 06:17:17,930 
2023-03-29 06:17:17,931 Processing full data for acis2eng
2023-03-29 06:17:18,102 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/full.pkl.gz
2023-03-29 06:17:18,108   elapsed time: 0.006 sec
2023-03-29 06:17:18,110 Applying updates to 105 h5 files
2023-03-29 06:17:18,840   elapsed time: 0.730 sec
2023-03-29 06:17:18,841 Updating /Users/aldcroft/ska/data/eng_archive/data/acis2eng/archfiles.db3
2023-03-29 06:17:18,854   elapsed time: 0.013 sec
2023-03-29 06:17:18,855 
2023-03-29 06:17:18,855 Processing 5min data for acis2eng
2023-03-29 06:17:19,024 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/5min.pkl.gz
2023-03-29 06:17:19,028   elapsed time: 0.004 sec
2023-03-29 06:17:19,031 Applying updates to 101 h5 files
2023-03-29 06:17:19,925   elapsed time: 0.894 sec
2023-03-29 06:17:19,926 
2023-03-29 06:17:19,926 Processing daily data for acis2eng
2023-03-29 06:17:20,045 Reading update date file https://icxc.cfa.harvard.edu/aspect/cheta/sync/acis2eng/2023-03-28T0245z/daily.pkl.gz
2023-03-29 06:17:20,045   elapsed time: 0.000 sec
taldcroft commented 1 year ago

What is the thought on the priority of adding a test to do "updates" that would have caught this?

There is some merit in having the test do a couple of updates to run through code that works on an existing sync archive. Maybe if I'm in that code again and remember then it'll happen but otherwise I'm not planning to prioritize it now.