VMA production - files fetched to wrong interface

[ahafele]

Metadata staff are working on setting up a number of interfaces to fetch files from OCLC WMS. Previously implemented was #915 to allow for connection using wms ssh key. We noticed on production that files with incorrect names were fetched to WMS interfaces. I am testing AIP update interface on stage and selected "process now". The DAG runs and I receive an email that files were fetched but the link in the email takes me to Springer interface (also WMS) where my AIP file has in fact been fetched.

Similarly on production an email for https://sul-libsys-airflow-prod.stanford.edu/vendor_management/interfaces/128 links to https://sul-libsys-airflow-prod.stanford.edu/vendor_management/interfaces/109 where I find the files.

[ahafele]

Files are also failing processing https://sul-libsys-airflow-stage.stanford.edu/log?execution_date=2024-05-08T13%3A00%3A04.756246%2B00%3A00&task_id=setup&dag_id=default_data_processor&map_index=-1

[jermnelson]

Hi @ahafele, on stage (and in production) both AIP and SPRINGER have the same interface UUID of e5d49284-3228-4556-bf5e-9b4c61535fc7 which I think is causing the first problem because of a subtle bug in how the ftp_download_task is called.

When the DAG is initiated from the AIP Interface Process Now button we are sending the following parameters to data_fetcher DAG:

• vendor_interface_name: WMS
• vendor_code: AIP-SUL
• vendor_uuid: fd40d6ae-74f0-5156-9ac3-5d0c1e4949c6
• vendor_interface_uuid: e5d49284-3228-4556-bf5e-9b4c61535fc7
• dataload_profile_uuid: None
• remote_path: /xfer/metacoll/out/ongoing/updates
• filename_regex: ^metacoll.STF.+AIP.1.mrc$
• download_path: /opt/airflow/vendor-data/downloads/fd40d6ae-74f0-5156-9ac3-5d0c1e4949c6/e5d49284-3228-4556-bf5e-9b4c61535fc7
• environment: stage

In the ftp_download_task we are querying the database by the Interface UUID which returns the SPRINGER interface (because it was added to the database before the AIP interface, you can tell because the id is 171 verses the AIP interface id of 178) BUT because the other values from the parameters are from the AIP interface in VMA, like the filename_regex, remote_path, and download_path, the download succeeds but the resulting VendorFiles like 19378 are now associated with the SPRINGER interface and not the AIP interface.

I can see in the FOLIO interface how you can share interfaces between organizations like AIP and SPRINGER but I think VMA assumes they will be different.

[ahafele]

Thanks for investigating this, Jeremy. I know we talked about this specific OCLC/Shared interface use case during the initial VMA work cycle but I don't know if/how it was accounted for - sounds like it wasn't. Is there anything we can change in VMA to allow for shared interfaces in FOLIO? There is nothing limiting us from creating new interfaces in FOLIO but it will result in many with the same connection information. I think somewhere around 45 if I had to guess.

[jermnelson]

Thanks for investigating this, Jeremy. I know we talked about this specific OCLC/Shared interface use case during the initial VMA work cycle but I don't know if/how it was accounted for - sounds like it wasn't. Is there anything we can change in VMA to allow for shared interfaces in FOLIO? There is nothing limiting us from creating new interfaces in FOLIO but it will result in many with the same connection information. I think somewhere around 45 if I had to guess.

I've started work on a VMA fix now

[ahafele]

@jermnelson on airflow stage I ran AIP and it fetches the correct file but the DAG shows as failing the archive task. The files do not get processed. The email I received still links to the Springer Interface >WMS (AIP-SUL) - e5d49284-3228-4556-bf5e-9b4c61535fc7

I ran a test for Springer interface and that is working as expected - files are fetched and processed, email is received with correct link.

[ahafele]

@jermnelson I ran AIP and the data_fetcher dag is now successful. The email is still incorrect. I will forward email and including content below. Filename is correct but references springer interface in text and subject and links to springer interface. data_processor dag is successful but the processed files don't appear in the UI and their status remains "loading"

Email content

WMS (SPRING-SUL) - e5d49284-3228-4556-bf5e-9b4c61535fc7

File processed, but not loaded: metacoll.STF.updates.D20240507.T213015.AIP.1.mrc

[jermnelson]

Hi @ahafele, looking through the emails, I'm wondering if multiple emails are being generated. I found this email from 5:56 am (PDT) that seems correct for the AIP and looks like it was generated from this DAG run. Here is the screen-shot of the email:

At 6:00 AM (PDT) the following email was sent (that you included above) with SPRINGER vender ocde but having the first of the two files from the AIP DAG run:

At 6:05 AM (PDT) a second email was sent with the SPRINGER vendor for the second file:

The latter two emails were generated and sent well after the AIP DAG run was finished at 5:55 PDT. These two follow-up emails come from default_data_processor DAG run that sends an email and looks related to the error you've noted about the processed files not showing up in the UI. I think the best approach is to get this PR approved and merged and create a follow-up ticket to fix the default_data_processor DAG, where it looks the code doesn't account for shared WMS vendor interfaces among different vendors.

[ahafele]

Hi Jeremy, That sounds good. One thing to note - in the first email (first screen shot above) the link still goes to the Springer interface.

[ahafele]

@jermnelson putting testing results here but let me know if a new ticket is preferred.

1. Fetch email no longer includes interface link and no longer lists the files fetched.

Prod

Stage

3. File processed email is missing Interface name and "Stage"

Prod

Stage

I see some failed file_loaded_email tasks e.g. here.

5. AIP files are still not being processed. https://sul-libsys-airflow-stage.stanford.edu/vendor_management/interfaces/178 They now have a status of fetched loading but are past the expected processing time and don't show processed files.