Retry tables open file

[taldcroft]

Description

Use https://github.com/sot/ska_helpers/pull/21 to hopefully make the daily archive updates more resilient to HEAD NetApp issues.

2021-02-08 04:03:57,089 Updating stats file /proj/sot/ska3/flight/data/eng_archive/data/acisdeahk/daily/TMP_FEP1_MONG.h5
2021-02-08 04:03:57,175   Adding 3 records
2021-02-08 04:03:57,211 Updating stats file /proj/sot/ska3/flight/data/eng_archive/data/acisdeahk/5min/TMP_FEP1_MONG.h5
Traceback (most recent call last):
  File "/proj/sot/ska3/flight/bin/cheta_update_server_archive", line 33, in <module>
    sys.exit(load_entry_point('Ska.engarchive==4.51.0', 'console_scripts', 'cheta_update_server_archive')())
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/cheta/update_archive.py", line 1186, in main
    main_loop()
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/cheta/update_archive.py", line 266, in main_loop
    update_stats(colname, '5min', msid)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/cheta/update_archive.py", line 507, in update_stats
    stats = tables.open_file(stats_file, mode='a',
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/tables/file.py", line 315, in open_file
    return File(filename, mode, title, root_uep, filters, **kwargs)
  File "/proj/sot/ska3/flight/lib/python3.8/site-packages/tables/file.py", line 778, in __init__
    self._g_new(filename, mode, **params)
  File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace

  File "H5F.c", line 509, in H5Fopen
    unable to open file
  File "H5Fint.c", line 1400, in H5F__open
    unable to open file
  File "H5Fint.c", line 1615, in H5F_open
    unable to lock the file
  File "H5FD.c", line 1640, in H5FD_lock
    driver lock request failed
  File "H5FDsec2.c", line 941, in H5FD_sec2_lock
    unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'

End of HDF5 error back trace

Unable to open/create file '/proj/sot/ska3/flight/data/eng_archive/data/acisdeahk/5min/TMP_FEP1_MONG.h5'

Testing

• [N/A] Passes unit tests

• [x] Functional testing

• Updated Ska.engarchive.update_archive to use this new function in every place that an HDF5 file is opened (substitute tables_open_file for tables.open_file.

• Make a new conda env on HEAD

• Install this package and Ska.engarchive into the environment using python setup.py install --single-version-externally-managed --record=record.txt

• Go to ska_testr repo and run run_testr --include Ska.engarchive. This runs the long test that exercises update_archive and this new function. PASS.