Test Workflows after back to pip migration

[ErnestaP]

Tested, everything looks working as before

[ErnestaP]

Workflows failed task brief analysis Cannot check ALL of the crashed tasks, because we don't have logs from 2022 till early 2023. Need tp check why and how to configure it.

Errors are listed by distinguishing every DAG separately. However, all of the mentioned errors can appear in ALL dags.

1. springer_process_file: Add conditions for extracting text from nodes, in order to avoid errors like this: ('NoneType' object has no attribute 'text'; 919). Also, missing default values, where the value is not found for the errors like this: TypeError: 'NoneType' object is not iterable, see them in Springer, but need to check all parsers, just in case.
   3. aps_process_file: Does the whole workflow has to stop and crash (not to save the record in db) if we cannot get arxiv categories? For example:Error: requests.get("http://export.arxiv.org/api/query", params)``requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
   4. hindawi_fetch_api: KNOWN issue: sometimes we cannot get a record from Hindawi: error
   5. KNOWN issue: IOP sends records with duplicated affiliation values, so the record cannot pass validation: error : [{'country': 'Italy', 'institution': '(A)University of Turin, I-10125, Turin, Italy; ' '(B)University of Eastern Piedmont, I-15121, ' 'Alessandria, Italy; (C)INFN, I-10125, Italy'}, {'country': 'Italy', 'institution': '(A)University of Turin, I-10125, Turin, Italy; ' '(B)University of Eastern Piedmont, I-15121, ' 'Alessandria, Italy; (C)INFN, I-10125, Italy'}]
   6. iop_process_file: IOP puts the month as a string, for example: November. We try to parse it to int and the workflow crashes. Need to add mapping for months(?): error, doi: 10.1088/1674-1137/ac90af
   7. oup_process_file: OUP sent the article without license: error, I checked XML content in our s3 bucket as wel. In production the record is correct- we have the same article uploaded. Most likely they uploaded the update.
   8. oup_pull_ftp: OUP, recently didn't happen: ftplib error: terminates the connection
   9. springer_process_file: Springer sent the articles without titles: just one of the examples, there are more than one. Maybe we should save somewhere the articles' dois which have the problems coming from the publishers' side (for example missing title field) and the reason for the crashed workflows? Will be easier to inform publishers with existing problems. Because now, we need to go and check in every run, then in the task, which crashed, to see the error. error
   10. springer_pull_ftp: Springer while connecting to sftp server, recently didn't happen: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer , error
   11. Check: do all of the DAGs has the same running interval as in the prod?