Installation on Ubuntu 20.04

[carlostighe]

Running in a virtual environment. Set PANDEM_HOME to ps-data The absolute path of ps-data is /home/user/sites/pandem2/pandem-source/ps-data

In storage.py line 76

def read_file(self, path): #absolute path here
    if path.startswith('PANDEM_HOME'):
        file_path = path
    else :
        file_path = os.path.join(os.getenv('PANDEM_HOME'), 'files', path)
    if file_path.split('.')[-1] == "json":
        with open(file_path, 'r') as f:
            data_dict = json.load(f)
        return data_dict
    else :
        with open(file_path, 'rb') as f:
            bytes_data = BytesIO(f.read())
        return bytes_data

the path being passed in is correct ps-data/files/source-definitions/medisys.json but its not the absolute path as the comment might suggest. pandem home is being read correctly as - ps-data/
Its hitting the else section of this function and appending ps-data/files to the path leaving the path looking like this - ps-data/files/ps-data/files/source-definitions/medisys.json
Wondering is this a virtual environment issues. Going to test setting PANDEM_HOME as PANDEM_HOME

[carlostighe]

changed PANDEM_HOME to PANDEM_HOME
process gets further along.

R script will be runned with: PANDEM_HOME/files/scripts/R/download-covid19-datahub.R
Fatal error: cannot open file 'PANDEM_HOME/files/scripts/R/download-covid19-datahub.R': No such file or directory
R script has just been run

The file mentioned above in the fatal error does exist. The process moves beyond this stage.

Run into another file path error.

2022-05-11 13:14:46,773 - pykka - INFO - Exception returned from Variables (urn:uuid:9ba8e196-c960-4ae5-8134-92cf6280e54e) to caller:
Traceback (most recent call last):
  File "/home/carlos/.virtualenvs/pandem-source/lib/python3.8/site-packages/pykka/_actor.py", line 186, in _actor_loop
    response = self._handle_receive(envelope.message)
  File "/home/carlos/.virtualenvs/pandem-source/lib/python3.8/site-packages/pykka/_actor.py", line 286, in _handle_receive
    return callee(*message.args, **message.kwargs)
  File "/home/carlos/sites/pandem2/pandem-source/pandemsource/variables.py", line 164, in write_variable
    self.write_variable(self.tag_source_var(job["dls_json"]), None, job)
  File "/home/carlos/sites/pandem2/pandem-source/pandemsource/variables.py", line 219, in write_variable
    with open(file_path, 'w+') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'PANDEM_HOME/PANDEM_HOME/files/variables/tag_source/default.json'

again it seems to be appending an extra directory onto the path.

After that there is a continual output along the lines of:

2022-05-11 13:15:28,173 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 6 source ecdc-covid19-variants 40.0%
2022-05-11 13:15:28,255 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 40.0%
2022-05-11 13:15:28,255 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 2 source geonames-countries 40.0%
2022-05-11 13:15:28,255 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 5 source ecdc-atlas-influenza 40.0%
2022-05-11 13:15:28,255 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 6 source ecdc-covid19-variants 40.0%
2022-05-11 13:15:28,293 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 45.4%
2022-05-11 13:15:28,312 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 45.8%
2022-05-11 13:15:28,328 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 46.4%
2022-05-11 13:15:28,348 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 47.0%
2022-05-11 13:15:28,367 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 47.8%
2022-05-11 13:15:28,375 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 48.8%
2022-05-11 13:15:28,497 - pandem.pipeline - DEBUG - Source influenza-net has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 13:15:28,497 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 4 source influenza-net 40.0%
2022-05-11 13:15:29,160 - pandem.pipeline - DEBUG - Source geonames-countries has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 13:15:29,160 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 2 source geonames-countries 40.0%
2022-05-11 13:15:29,166 - pandem.pipeline - DEBUG - Source ecdc-atlas-influenza has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 13:15:29,166 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 5 source ecdc-atlas-influenza 40.0%
2022-05-11 13:15:29,174 - pandem.pipeline - DEBUG - Source ecdc-covid19-variants has been reversed to read format end since some referential failed completely, probably some depending source is missing  

[carlostighe]

restarting after interrupt ctrl + c leads to the following

2022-05-11 13:22:51,755 - pykka - ERROR - Unhandled exception in Storage (urn:uuid:14914e2a-ff75-4358-a078-c44139e97ee4):
Traceback (most recent call last):
  File "/home/carlos/.virtualenvs/pandem-source/lib/python3.8/site-packages/pykka/_actor.py", line 179, in _actor_loop
    self.on_start()
  File "/home/carlos/sites/pandem2/pandem-source/pandemsource/storage.py", line 47, in on_start
    self.db_tables['issue'] = pd.read_pickle(os.path.join(os.getenv('PANDEM_HOME'), 'database/issues.pickle'))
  File "/home/carlos/.virtualenvs/pandem-source/lib/python3.8/site-packages/pandas/io/pickle.py", line 205, in read_pickle
    return pickle.load(handles.handle)
EOFError: Ran out of input

pykka._exceptions.ActorDeadError: Storage (urn:uuid:14914e2a-ff75-4358-a078-c44139e97ee4) not found
pykka._exceptions.ActorDeadError: Orchestration (urn:uuid:e275ba3d-511e-45e5-a276-825be87d1b0e) stopped before handling the message

[carlostighe]

Working around the filepath issues and left with these errors:

Source influenza-net has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 16:02:24,659 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 4 source influenza-net 40.0%
2022-05-11 16:02:24,822 - pandem.pipeline - DEBUG - Source geonames-countries has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 16:02:24,822 - pandem.pipeline - DEBUG - Changing to step read_df_ended for job 2 source geonames-countries 40.0%
2022-05-11 16:02:25,392 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 5 source ecdc-atlas-influenza 40.0%
2022-05-11 16:02:25,392 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 4 source influenza-net 40.0%
2022-05-11 16:02:25,393 - pandem.pipeline - DEBUG - Changing to step standardize_started for job 2 source geonames-countries 40.0%
2022-05-11 16:02:33,585 - pandem.pipeline - DEBUG - Source ecdc-covid19-variants has been reversed to read format end since some referential failed completely, probably some depending source is missing
2022-05-11 16:02:33,834 - pandem.pipeline - DEBUG - Source ecdc-atlas-influenza has been reversed to read format end since some referential failed completely, probably some depending source is missing

pandem.pipeline is continually running trying to finish the process but running into the same errors.

Going to the Data Sources page and I get the below The errors is my terminal arent of any use to me in trying to debug that?