issue when dealing with encoding error in exotethys core function

[mariomorvan]

python version: 3.7 and 3.10 pylightcurve branch: dev or master on 11 May 2023

Calling plc.exotethys raises a series of errors (UnpicklingError, UnicodeDecodeError, TypeError) where the last one is:

~/git-repos/pylightcurve/pylightcurve/processes/files.py in open_dict(path)
     44             return unpickler.load()
     45         except UnicodeDecodeError:
---> 46             return unpickler.load(encoding='latin-1')
     47 
     48 def save_dict(dictionary, path):

TypeError: load() got an unexpected keyword argument 'encoding'

Checking at the various versions (back to 3.7) of the _Unpickler object it does not seem that the load method accepts the encoding argument.

[gmorello]

The encoding argument in exotethys was for compatibility with python2. That part of code occur because it failed to read the pickle file as by default (except UnicodeDecodeError). The most common case in which this happens is that one of the downloaded database files is corrupted, due to e.g. poor wifi connection, special security settings, or others. If a downloaded file is corrupted, you must remove it using the manage_database subpackage of exotethys. We should discuss how to avoid that the same error occurs again during the download.

[mariomorvan]

Thank you Giuseppe for the insight. If the issue is due to poor connection during download, would it be a good idea to provide an option to delete the corrupted files? Or do it automatically in the case of failure?

[atsiaras]

Hi both, in the dev brunch I have added an automatic check which will download the files again if a problem has occurred during downloading. The encoding='latin-1' options should be removed. Thanks for pointing out!

[gmorello]

Hi Angelos, that sounds good, thanks. I will try to implement a similar turnaround directly in ExoTETHyS as well. Working on some updates right now.