Porting plugins from v2 to v3

[cleaaum]

When porting a plugin, it should be moved from the /plugin dir to the /flowcb dir. To group them together, thinking of putting all on_message plugins (which become after_accept) into its own file directory called accept.

Documentation written/refined as a result of this issue:

https://metpx.github.io/sarracenia/How2Guides/v2ToSr3.html

So far the following plugins have been ported: Note: some of these still have FIXMEs and TODOs to take care of since I wasn't always sure about the logic of the plugin

Make your branch's PR reference this issue....

Misc --> /flowcb

• Active Branch: v03_wip_plugins

• Waiting on sr3 implementation for the rest.. no touchy

  • [x] data_nocr.py ( @petersilva )

  • The on_data entry point is not supported at the moment. it has the same problem as on_html_data (#426), being called from the transfer/ classes rather than flow/

  • documented in eabfe7696be177f908b21b80d880ee2451f9bed6

  • [x] pxSender_log.py /( @petersilva )

  • The idea is to have sarra spew logs that look like PX so when people are porting PX->sarra they can log compare (vimdiff)

  • in v2, this is addressed by adding the --dry_run option to sender... ( #465 ) the dry_run/simulation mode is not yet implemented for sr3. Once that is done, this plugin is no longer needed.

  • [x] root_chown.py

  • [x] trace_on_stop.py

flowcb/destfn/

• [x] destfn_sample.py (#536) ... sample.py exists.
  • This will be interesting also... it is basically an after_accept scoped to apply to a single accept... dunno if that works properly in v3... it might. (@petersilve, addressed this with #550 )

do_send --> /flowcb/send

• Active Branch: v03_wip_plugins_v2tov3_send by @cleaaum
  • [x] do_send_log.py

on_heartbeat --> /flowcb/housekeeping

• Active Branch: v03_wip_plugins_v2tov3_hb by @MagikEh
• [x] hb_memory.py
• [x] hb_police_queues.py @petersilva says it's ok to drop this. It isn't clear this is actually used anywhere... and it deletes queues, which seems draconian... functionality is just missing in sr3... could always re-create later.

on_message --> /flowcb/accept

• Active Branch: v03_wip_plugins_v2tov3_msg by @cleaaum
  • [x] msg_2http.py
  • [x] msg_2local.py
  • [x] msg_2localfile.py
  • [x] msg_WMO_type_suffix.py
  • [x] msg_delay.py
    • yes/no/maybe... look at ../flowcb/mdelaylatest and see if can be merged.
  • [x] msg_delete.py
  • [x] msg_download_baseurl.py
    • test with nexrad stuff, nexrad is already ported... have to investigate to understand.
  • [x] msg_filter_wmo2msc.py
    • should be called flowcb/filter/wmo2msc.py
  • [x] msg_hour_tree.py
• [x] msg_ #434
  • part of Michel Grenier's porting? low priority... little applicability... requires discussion.
    • [x] msg_http_to_https.py
    • [x] msg_print_lag.py
    • best done by adding options to log, to include stats, perhaps in housekeeping.
    • did the porting to v3 (Clea). So maybe we delete this one later and modify log instead..
    • [x] msg_rename4jicc.py
    • [x] msg_rename_dmf.py
    • [x] msg_rename_whatfn.py
    • [x] msg_renamer.py
    • [x] msg_replace_new_dir.py @petersilva ... ( 5e9e1c02cc9a15150a600c7895fb39bb930c7a32 )
    • (required for flow tests)
    • [x] msg_save.py
    • [x] msg_skip_old.py
    • [x] msg_speedo.py
    • [x] msg_sundew_pxroute.py
    • [x] msg_test_retry.py
    • [x] msg_to_clusters.py
    • [x] post_hour_tree.py
    • [x] post_long_flow.py
    • For testing the max length of ampq messages (256 characters)
    • [x] post_override.py

on_file --> /flowcb/after_work

• Active Branch: v03_wip_plugins_v2tov3_file by @cleaaum
  • [x] on_file_repair_size.py
  • [x] file_age.py ... ( @cleaaum made a first pass, but it isn't done... It is not really ported, I finished porting it (changing the name to flowcb/work/age.py... but even then, there is a conceptual problem. It is supposed to write in the log how far a subscribe is lagging behind a source, but the numbers are going to be put way off by batching. opened a new issue for that #561 )
  • [x] file_check.py
  • [x] file_citypage_check.py
  • [x] file_email_decode.py
  • [x ] file_mail_metareas.py ... never found this one... ignoring for now.
  • DOR for INMARSAT
  • [x] file_send_egc_les.py
  • file sender for INMARSAT ... to be replaced by new API
  • [x] part_clamav_scan.py
  • Move up to file_av_scan.py

do_poll--> /flowcb/poll/

• [x] poll_noaa.py
• [x ] poll_script.py -> flowcb/run.py
• [x] poll_shc_soap.py
• [x] poll_usgs.py
• [x] download_email_ingest.py --> flowcb/download/mail_ingest.py
• [x] download_shc_soap.py (@petersilva this is done, called flowcb/poll/noaa_hydrometric.py )

Plugins Deleted So Far

download_ --> /flowcb//

• [x ] exp_2mqtt.py (was a bridge to MQTT, now have full mqtt support.)
• [x] smc_download_cp.py (irrelevant in sr3 because getAccellerated in transfer classes... aka built-in.)
• [x] download_noaa.py
  • check with ops if newer version there. perhaps do_download is still missing... not sure... this one could be interesting. -PS
• [x] download_usgs.py
  • need to look at flowcb/poll/usgs.py and see if this is still needed. not sure what it does. -PS
• [x] msg_from_cluster.py
  • [x] To be deleted/depreciated
• [x] harness.py
• [x] hb_retry.py
  • Included in /flowcb/retry.py
• [x] hb_message.py -Measures lagg, should be folded into ../log.py
• [x] msg_stdfiles.py
• [x] msg_fdelay.py
• [x] msg_stopper.py
• [x] msg_overwrite_sum.py
• [x] msg_gts2wistopic.py
• [x] msg_download.py
• [x] msg_by_source.py
• [x] msg_by_user.py
• [x] msg_dump.py
  • no function provided by flowcb/log.py
• [x] msg_auditflow.py (@petersilva looked at it and killed it.)
  • requires discussion... seems flow_ test related. not sure what it does.
• [x] line_log.py
  • [x] The documentation at the top seems wrong. It is a copy-paste of the description of the line_mode plugin.
• [x] line_mode.py
  • The functionality of this plugin is now included in the line_to_SFTPattributes plugin, and can be deleted for v3 (Clea).
• [x] part_check.py
  • Check if it matches file_check.py
• [x] part_file_assemble.py
  • This is part of partitioning. So dead for now. lowest priority until this feature is re-enabled.
• [x] hb_pulse.py
  • not needed, implemented in sr3
• [x] hb_sanity.py
  • not needed, sr3 sanity is run as a cronjob now
• [x] html_page.py
• [x] msg_total.py
  • functionality added into flowcb/log.py
• [x] msg_total_save.py
  • functionality added into flowcb/log.py
• [x] post_total.py
  • functionality added into flowcb/log.py
• [x] post_total_save.py
  • functionality added into flowcb/log.py
• [x] file_total.py
  • functionality added into flowcb/log.py
• [x] file_total_save.py
  • functionality added into flowcb/log.py
• [x] msg_cclean_f90.py
  • cclean is not used anywhere. it's a left-over.
• [x] msg_cclean_f91.py
• [x] msg_cclean_f92.py

[petersilva]

do_poll -> flowcb/poll/ check in the poll directory as I think most of the examples are already done.

[cleaaum]

Question 1: what is the difference between the action plugins and the filter plugins since they both are after_accept() plugins? Does it make sense to keep them in separate directories? the init .py of filter states "plugins intended for on_message entry_point." Seems like they could all be moved to action.

[cleaaum]

Question 2: some plugins like totalsave.py have on_start() and on_stop() methods. Should they me modified or renamed for v3?

[petersilva]

merged in all current work from the v03_wip_plugins_v2Tov3_msg branch. also updated that branch with v03_wip... to reduce divergence as people work on it, people should pull from v03_wip once in a while...

it passed flows.

[tysonkaufmann]

file/msg/post total plugins are now obsolete and the functionality has been added directly into flowcb/log.py - plugins were deleted and config.py updated to reflect deleted plugins on branch v03_wip_plugins_v2Tov3_msg

Branch v03_wip_plugins_v2Tov3_msg will need to be merged into v03_wip @petersilva

[MagikEh]

Went through the project #6's notes in the PluginsToPort column and merged their comments into this issue's description, as this issue is better for tracking.

[MagikEh]

Moved from project #6 to the Metpx Project.

[petersilva]

summarizing recent work (last day or two):

• ported msg_replace_new_dir to sr3 ... also added to config convert logic.
• deleted msg_auditflow.py
• figured out download_shc_soap.py is already ported to sr3.
• documented how to do on_data plugins (data_nocr.py)
• documented what is going on with pxsender_log.py
• I have confirmed that cclean is not used anywhere... deleted them.

These are all documented by direct editing of the summary at the top of the issue.

[petersilva]

We have gotten to the point where there functionality is good enough to start deploying to internal clusters. so the focus is on that for now (will be tracked by internal tickets.) May come back to this at some point.

[petersilva]

Internally have identified 234 plugins to port.

[petersilva]

using two phase approach, phase 1: initial port. phase 2: testing/deployment.

in phase 1: 17/232 done (two "plugins" turned out to be shell scripts)

[petersilva]

29/232 done.

[petersilva]

56/236 done. (removed some, found some new ones due to continued audits.)

[petersilva]

83/236 done. All polls completed, a few downloads done. downloads and converters are next target.

[petersilva]

Features brought into sr3 based on analysis of plugins:

• nodupe.v2wrapper.sumstrFromMessage() created to simplify ECCC sumstr_ext plugin.
• adding interpretation of an inflight duration (and more explicit nodupe_fileAgeMin ) makes minimum age usable as an acceptance criteria for all components (formerly was not being honoured for polls.) allows elimination of line_rvas_wait.py plugin, since this was it's only reason for existence.

update: 154/236 done... 65% !

• all cases of destfn (DESTFNSCRIPT) done.

[petersilva]

Devs want us to strip out the examples from the code.

• discussing making a MetPX/sr3-examples github repo that contains the examples from the main repo
• publish a reviewed subset of the internal repo of plugins... worry about inadvertant disclosure,
• for sr3 list examples would need to add an, example path of sorts to search for additional ... or just de-commission "add" and have people explicitly copy files...
• or "sr3 list examples", gives download advice... and once downloaded, works normally...
• strip out list examples...

let documentation take care of it.

[petersilva]

163/236 -> 69% (mile stone: all post_ completed.)

[petersilva]

178/236 -> 75%

[andreleblanc11]

185/236 , only 6 converts remain

[petersilva]

218/235, all the on_message renames are done.

[petersilva]

228/235... started the config conversion script.

[andreleblanc11]

235/235. All done! 😄

[petersilva]

removed plugins covered by 2f2878d38d5e1e0ea99554efea030c30f00dff5a last goal is to review the few remaining ones in that directory... to empty sarracenia/plugins/ (where v2 plugins live in sr3)

[petersilva]

completely done!