Have the message `rename` field be filterable

[andreleblanc11]

Use case

• On dev, we get messages from the DMS winnows. Those messages include the rename field in the message. The logs from the winnow show that the rename field is set when the winnow posts.

winnow_dms-normal_01.log:2024-06-24 14:28:38,003 [INFO] sarracenia.flowcb.log after_accept accepted: (lag: 0.15 ) http://server:8180 /data/dms/notification/environment/hazard/wildfire-1.0-ascii/product-notification_ascii-1.0/20240624142800000/canada/all
winnow_dms-normal_01.log:2024-06-24 14:28:38,005 [INFO] sarracenia.flowcb.log after_post posted to exchange: xs_MSC-DMS-DEV topic: v02.post.data.dms.notification.environment.hazard.wildfire-1.0-ascii.product-notification_ascii-1.0.20240624142800000.canada a file with baseUrl: http://server:8180 relPath: data/dms/notification/environment/hazard/wildfire-1.0-ascii/product-notification_ascii-1.0/20240624142800000/canada/all rename: WILDFIRE_NOTIF_20240624142800000_CANADA_ALL

• When the message reaches the sarra on ddsr dev, the current sr3 filtering won't filter based on the message rename field. It would be nice if we could have the filtering be able to filter based on the renamed filename.
• Myself and @reidsunderland were thinking of adding an elif after the sundew extension (see snippet below), where we could change the urlToMatch based on the renamed file.

https://github.com/MetPX/sarracenia/blob/73fdf9f8bac1844534312c5c625312178e1ea273/sarracenia/flow/__init__.py#L1032-L1035

• I also have to admit that seeing the rename: new-filename inside of the winnow log did confuse me. I spent a bunch of time futzing around with the accept statement of the sarra component before realizing that the filename doesn't change when arriving to the sarra so another improvement could be improving the logging to avoid this confusion.

[petersilva]

thinking:

• until today, filtering is on the upstream url alone. It doesn't look at any downstream values or results. so when formulating accept/reject you can just look at the where the files are on the upstream server to know what to filter on. The "rename" headers are not visible in the upstream... the user has to download a few...look at the message, notice the rename, and then start filtering? Is that the expectation?

• if you have plugins (which have the name... you may have noticed "after_accept") all the renamers take effect after the accept/reject filtering has happenned... so it's confusing that if using the rename header in the message, the final name is used, but the after_accept stuff is totally ignored... should we then have "before_accept" make a class of renamers that work for that case?

• the rename header is kind of a left-over of early development... not really meant to be used going forward... I know DMS uses it. the more future oriented way of accomplishing that is to use relpath and RetrievePath, and then the normal filtering will work:

  • have the current relPath be assigned to retrievePath instead.
  • have the relPath be the name & path we want to download as.

  Maybe we can implement a filter or internal logic to do that all the time....have not thought about it enough to say for sure. This could certainly be implemented using a before_accept() plugin, if we made those...

• Currently filtering happens on one path, if we filter on two paths, it brings up the possibility of conflict:

  • what should happen if the rename is rejected but the main relpath is accepted?
  • what should happen if rename is accepted, but the relPath is rejected?

Given that you are getting the messages from the DMS winnows., you could implement an after_accept on the winnows themselves that swap the rename header for relPath and retrievePath instead... they might have to publish as v3 though... not sure if retrievePath is a thing in v2... would have to try out. anyways... if that were done, the current default processing would work on the downstream consumers with no issue. so there would be no core code change then, just a new plugin. not saying we should do that... just trying to tease out what options there are.

[petersilva]

another interpretation... (maybe this is what you meant? )

https://github.com/MetPX/sarracenia/blob/73fdf9f8bac1844534312c5c625312178e1ea273/sarracenia/flow/__init__.py#L1032-L1035

• in the code snippet given... look a few lines higher... url is set... but it is perhaps set wrong... url could be evaluated such that the rename field takes effect there... then the sundew_extension can be added as normal...

https://github.com/MetPX/sarracenia/blob/73fdf9f8bac1844534312c5c625312178e1ea273/sarracenia/flow/__init__.py#L1024-L1030

• In this case... This will be a change of behavior... I dunno if it is desirable or not... it's different from how rename has always worked. but in this case there is only one url evaluated, not two, it is just the post rename one... I don't know if other sources are using rename as well.. would have to analyze.

I think it would be good to try adding an after_accept plugin to the winnows to map the relPath to retrievePath, and have the relPath refer to renamed version (as described in previous post).... and see how it goes... It's a good approach because it will only affect the use case being raised, does not change things for anyone who depends on the current behaviour.