Enable and test publishers on QA

[drjova]

Make sure that the workflows are running for the following publishers:

• [ ] APS issue
• [ ] HINDAWI issues
• [ ] SPRINGER: issue

Airflow Checklist

• Check if the publishers are enabled
• Check and enable if is missing the schedule for its publisher, it should be the same as in PRODUCTION
• Are the workflows triggered correctly? Are any errors? What happens if there are no new files from the publisher _Dags for pulling_from a source (sftp or API) has to be triggered by the schedule. The dags responsible for processing files, have to be triggered by the output of pulling_from a source DAG. The source file is split and every record triggers its own run. If the pullingfrom a source DAG doesn't pull any files, the DAGs last task finishes in success state and no new runs in processing DAG are triggered.

Record Checklist

• Compare the workflows QA records with the production one (it should miss only the _files field). The records from workflows on QA should be valid records.
• Are any fields missing or malformed? issue
• Are any errors?
• Are the logs clear? issue

General change:

• Make s3 file structure the same as in sftp
• Move to a different library: from pysftp to paramiko: issue

[ErnestaP]

Update:

• ~~argocd: couldn't find key IOP_FTP_USERNAME in Secret scoap3-qa/scoap3-workflows-iop-creds Tried to fix it link, but unsuccessfully, I will ask Ben.~~, fixed

• ~About DAGs: they are scheduled to run every 3-4 days. However, need to fix the file processing Dags, so that they would run just on the trigger~

• ~Hindawi api is crashing: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://www.hindawi.com/oai-pmh/oai.aspx?from=2022-12-13&until=2023-01-18&verb=listrecords&set=HINDAWI.AHEP&metadataprefix=marc21~ Not on our side

• springer_pull_ftp: raise SSHException( paramiko.ssh_exception.SSHException: Unable to connect to scoap3-workflows-sftp: [Errno 110] Connection timed out