search

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows
Apache License 2.0
5 stars 0 forks source link

VMA - files not found for interface #944

Closed ahafele closed 4 months ago

ahafele commented 4 months ago

Interface has been successfully retrieving files until beginning of April. Interface configuration has not changed, DAG says 0 files found. Regex is correct.

Interface - https://sul-libsys-airflow-prod.stanford.edu/vendor_management/interfaces/43 Should be retrieving the following - -rw-r--r-- 1 incabook incabook 16334 Apr 10 09:41 ST26673.mrc -rw-r--r-- 1 incabook incabook 16016 Apr 10 09:41 ST26674.mrc -rw-r--r-- 1 incabook incabook 17279 Apr 10 09:41 ST26677.mrc -rw-r--r-- 1 incabook incabook 16446 Apr 10 09:41 ST26678.mrc -rw-r--r-- 1 incabook incabook 18469 Apr 24 15:36 ST26708.mrc -rw-r--r-- 1 incabook incabook 18349 Apr 24 15:36 ST26709.mrc -rw-r--r-- 1 incabook incabook 17779 Apr 24 15:36 ST26710.mrc

jermnelson commented 4 months ago

Looking at the Interface, I think the regular expression should be adjusted to ^Stanford/ST\d+.*\.mrc$, the "Stanford" directory is being prepended to the list of files so the regular expression is failing.

ahafele commented 4 months ago

Is this a change on our end and therefore across all interfaces? If so, most of our regexes will now be wrong and I wonder if we can revert back to how it behaved previously? Or maybe this is specific to this vendor, because I don't see it happening, e.g. for Casalini...

jermnelson commented 4 months ago

I don't think it is across all of the interfaces but it may help to check those interfaces that have a Remote Path value to see they are working as expected with their regex.

ahafele commented 4 months ago

I tried updating the interface with the suggested regex and got a download error.

jermnelson commented 4 months ago

@shelleydoljack found the following Stackoverflow answer that seems related to this ticket.