search

artefactual-sdps / preprocessing-sfa

preprocessing-sfa is an Enduro preprocessing workflow for SFA SIPs
1 stars 0 forks source link

Feature: make allow file format lists configurable #60

Closed sallain closed 1 month ago

sallain commented 1 month ago

Is your feature request related to a problem? Please describe.

The allow file format list is currently hardcoded. Anyone who wants to use this feature will have their own list of allowed formats, so they would have to change the code to suit their requirements. This also puts the list at risk during upgrades/redeployments as it may be reset to the hardcoded default.

Describe the solution you'd like

Make the allow file format list configurable.

In a short discussion with @djjuhasz, he suggested we could either use the enduro.toml config file, or by adding dedicated config files - with a dedicated config file, we could use CSVs instead of TOML, which would be more user-friendly. Open to any option, however!

Describe alternatives you've considered

Keep it as is and cry every time a user forgets to reset their list.

Additional context

djjuhasz commented 1 month ago

@sallain I moved this issue from the https://github.com/artefactual-sdps/temporal-activities repository because the "validate-file-format" activity is currently local to preprocessing-sfa. I do think we should move "validate-file-format" to the "temporal-activities" repo so it can be used in other Temporal projects, but it will be easier to do the configuration development here and then move the activity afterwards.

djjuhasz commented 1 month ago

@sallain we currently only have an "allowed file format" list — any format that is not on the list is disallowed. Should I add the ability to have a "disallowed file format" list as well? If we do add the disallow list, what should happen if a format is not on either list?

sallain commented 1 month ago

Thank you @djjuhasz!

No need for a disallow list at this time - I'll edit the issue. My bad.

sallain commented 3 weeks ago

@scollazo tested this with the following note: "I tested by removing Jp2000 from the list of allowed formats, launching a sip (it fails with the proper error/path), updating the list again, launching the same sip, and it works. no restart needed!"