search

MetPX / sarracenia

https://MetPX.github.io/sarracenia
GNU General Public License v2.0
45 stars 22 forks source link

Should retry'd transfers reflect updated configuration? #912

Closed reidsunderland closed 8 months ago

reidsunderland commented 8 months ago

Config before:

# oops, there's a typo:
directory /my_datta/
accept .*

If the problem in the config is fixed and the process is restarted:

directory /my_data/
accept .*

When sr3 starts up, failed messages will be recovered from the retry queue and their new_... fields will still have the original directory. This isn't wrong, but v2 works differently (I haven't looked at the code to prove it, but that's how I've seen it work).

We're considering deleting the new_ fields when recovering messages from retry queues.

If we don't change this, if a directory needs to be changed, the user would need to edit the retry queue file on disk. I assume it would be even harder to edit the contents of a retry queue stored in redis.

petersilva commented 8 months ago

I was told that this behaviour represents a regression vs. v2... please confirm.

reidsunderland commented 8 months ago

Testing with v2:

Config:

broker amqps://dd.weather.gc.ca/
topicPrefix v02.post

expire 1h

subtopic bulletins.alphanumeric.*.*.CWAO.#

directory /does_not_exist/
accept .*UANT01_CWAO.*
reject .*

Logs where downloads fail, files get added to retry queue:

$ sr_subscribe foreground test_change_dir
2024-01-31 18:13:10,207 [WARNING] extended option topicPrefix = ['v02.post'] (unknown or not declared)
2024-01-31 18:13:10,207 [INFO] sr_subscribe test_change_dir start
2024-01-31 18:13:10,208 [INFO] log settings start for sr_subscribe (version: 2.23.08):
2024-01-31 18:13:10,208 [INFO]  inflight=.tmp events=create|delete|link|modify use_pika=False topic_prefix=v02.post dry_run=False
2024-01-31 18:13:10,208 [INFO]  inline=False events=create|delete|link|modify use_amqplib=False topic_prefix=v02.post
2024-01-31 18:13:10,208 [INFO]  suppress_duplicates=False basis=path retry_mode=True retry_ttl=300000ms tls_rigour=normal
2024-01-31 18:13:10,208 [INFO]  expire=300000ms reset=False message_ttl=None prefetch=25 accept_unmatch=False delete=False poll_without_vip=True
2024-01-31 18:13:10,208 [INFO]  heartbeat=300 sanity_log_dead=450 default_mode=000 default_mode_dir=775 default_mode_log=600 discard=False durable=True
2024-01-31 18:13:10,208 [INFO]  declare_queue=True declare_exchange=True bind_queue=True
2024-01-31 18:13:10,208 [INFO]  post_on_start=True preserve_mode=True preserve_time=True realpath_post=False base_dir=None follow_symlinks=False
2024-01-31 18:13:10,208 [INFO]  mirror=False flatten=/ realpath_post=False strip=0 base_dir=None report_back=True log_reject=False
2024-01-31 18:13:10,209 [INFO]  Plugins configured:
2024-01-31 18:13:10,209 [INFO]          do_download:
2024-01-31 18:13:10,209 [INFO]          do_get     :
2024-01-31 18:13:10,209 [INFO]          on_message:
2024-01-31 18:13:10,209 [INFO]          on_data:
2024-01-31 18:13:10,209 [INFO]          on_part:
2024-01-31 18:13:10,209 [INFO]          on_file: File_Log
2024-01-31 18:13:10,209 [INFO]          on_post: Post_Log
2024-01-31 18:13:10,210 [INFO]          on_heartbeat: Hb_Log Hb_Memory RETRY
2024-01-31 18:13:10,210 [INFO]          on_report:
2024-01-31 18:13:10,210 [INFO]          on_start:
2024-01-31 18:13:10,210 [INFO]          on_stop:
2024-01-31 18:13:10,210 [INFO] log_settings end.
2024-01-31 18:13:10,210 [INFO] sr_subscribe run
2024-01-31 18:13:10,211 [INFO] AMQP  broker(dd.weather.gc.ca) user(anonymous) vhost(/)
2024-01-31 18:13:10,211 [INFO] Using amqp module (AMQP 0-9-1)
2024-01-31 18:13:10,231 [ERROR] standard queue name based on: prefix=q_anonymous program_name=sr_subscribe exchange_split=False no=-1
2024-01-31 18:13:10,233 [INFO] Binding queue q_anonymous.sr_subscribe.test_change_dir.49229110.20130665 with key v02.post.bulletins.alphanumeric.*.*.CWAO.# from exchange xpublic on broker amqps://anonymous@dd.weather.gc.ca/
2024-01-31 18:13:10,243 [INFO] declared queue q_anonymous.sr_subscribe.test_change_dir.49229110.20130665 (anonymous@dd.weather.gc.ca)
2024-01-31 18:13:10,247 [INFO] reading from to anonymous@dd.weather.gc.ca, exchange: xpublic
2024-01-31 18:13:10,249 [INFO] report_back to anonymous@dd.weather.gc.ca, exchange: xs_anonymous
2024-01-31 18:13:10,249 [INFO] sr_retry on_heartbeat
2024-01-31 18:13:10,254 [INFO] No retry in list
2024-01-31 18:13:10,256 [INFO] sr_retry on_heartbeat elapse 0.007346
2024-01-31 18:13:30,520 [WARNING] making /does_not_exist: [Errno 13] Permission denied: '/does_not_exist'
2024-01-31 18:13:30,520 [ERROR] could not cd to directory /does_not_exist
2024-01-31 18:13:30,520 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,522 [WARNING] downloading again, attempt 2
2024-01-31 18:13:30,523 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,524 [WARNING] downloading again, attempt 3
2024-01-31 18:13:30,525 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,527 [INFO] appended to retry list file 20240131181315.523 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___53188
2024-01-31 18:13:30,531 [WARNING] making /does_not_exist: [Errno 13] Permission denied: '/does_not_exist'
2024-01-31 18:13:30,531 [ERROR] could not cd to directory /does_not_exist
2024-01-31 18:13:30,531 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,533 [WARNING] downloading again, attempt 2
2024-01-31 18:13:30,533 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,535 [WARNING] downloading again, attempt 3
2024-01-31 18:13:30,535 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:30,537 [INFO] appended to retry list file 20240131181328.723 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___02502
2024-01-31 18:13:35,664 [WARNING] making /does_not_exist: [Errno 13] Permission denied: '/does_not_exist'
2024-01-31 18:13:35,664 [ERROR] could not cd to directory /does_not_exist
2024-01-31 18:13:35,664 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:35,666 [WARNING] downloading again, attempt 2
2024-01-31 18:13:35,667 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:35,668 [WARNING] downloading again, attempt 3
2024-01-31 18:13:35,668 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:35,670 [INFO] appended to retry list file 20240131181332.60 https://dd5.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___06373
2024-01-31 18:13:45,921 [WARNING] making /does_not_exist: [Errno 13] Permission denied: '/does_not_exist'
2024-01-31 18:13:45,921 [ERROR] could not cd to directory /does_not_exist
2024-01-31 18:13:45,921 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:45,923 [WARNING] downloading again, attempt 2
2024-01-31 18:13:45,923 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:45,925 [WARNING] downloading again, attempt 3
2024-01-31 18:13:45,926 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:13:45,927 [INFO] appended to retry list file 20240131181337.964 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___37295
2024-01-31 18:14:25,404 [WARNING] making /does_not_exist: [Errno 13] Permission denied: '/does_not_exist'
2024-01-31 18:14:25,404 [ERROR] could not cd to directory /does_not_exist
2024-01-31 18:14:25,404 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:14:25,407 [WARNING] downloading again, attempt 2
2024-01-31 18:14:25,407 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:14:25,409 [WARNING] downloading again, attempt 3
2024-01-31 18:14:25,409 [ERROR] sr_subscribe/__do_download__: Could not download
2024-01-31 18:14:25,411 [INFO] appended to retry list file 20240131181413.110 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311814___47827
^C2024-01-31 18:14:28,973 [INFO] signal stop (SIGINT)
2024-01-31 18:14:28,973 [INFO] sr_subscribe stop

Retry disk queue file:

$ cat ~/.cache/sarra/subscribe/test_change_dir/sr_subscribe_test_change_dir_00.retry.new
["v02.post.bulletins.alphanumeric.20240131.UA.CWAO.18", {"atime": "20240131181315.523", "filename": "msg_ddsr-WXO-DD3_6b510db3c95732769433f66db5173290:fromPxatx:CWAO:UA:3:Direct:20240131181312", "from_cluster": "DDSR.CMC", "mtime": "20240131181315.523", "parts": "1,125,1,0,0", "source": "WXO-DD", "sum": "d,051bdad5c7c408423873bf265a361490", "sundew_extension": "fromPxatx:CWAO:UA:3:Direct:20240131181312", "to_clusters": "DDSR.CMC,DDI.CMC,CMC,SCIENCE,EDM"}, "20240131181315.523 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___53188"]
["v02.post.bulletins.alphanumeric.20240131.UA.CWAO.18", {"atime": "20240131181328.723", "filename": "msg_ddsr-WXO-DD3_841bea2477d3f732ffec114b048deffb:fromPxatx:CWAO:UA:3:Direct:20240131181327", "from_cluster": "DDSR.CMC", "mtime": "20240131181328.723", "parts": "1,124,1,0,0", "source": "WXO-DD", "sum": "d,29ef6a74e946d1ea3c41df13d293acc7", "sundew_extension": "fromPxatx:CWAO:UA:3:Direct:20240131181327", "to_clusters": "DDSR.CMC,DDI.CMC,CMC,SCIENCE,EDM"}, "20240131181328.723 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___02502"]
["v02.post.bulletins.alphanumeric.20240131.UA.CWAO.18", {"atime": "20240131181332.60", "filename": "msg_ddsr-WXO-DD3_8700d326c8d05cc9f46490a5a0e5d0ad:fromPxatx:CWAO:UA:3:Direct:20240131181331", "from_cluster": "DDSR.CMC", "mtime": "20240131181332.60", "parts": "1,117,1,0,0", "source": "WXO-DD", "sum": "d,d5bd4dfd78325b991b3e66a083444530", "sundew_extension": "fromPxatx:CWAO:UA:3:Direct:20240131181331", "to_clusters": "DDSR.CMC,DDI.CMC,CMC,SCIENCE,EDM"}, "20240131181332.60 https://dd5.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___06373"]
["v02.post.bulletins.alphanumeric.20240131.UA.CWAO.18", {"atime": "20240131181337.964", "filename": "msg_ddsr-WXO-DD3_4e1d0ddcd4158100c6bb18110029c8c8:fromPxatx:CWAO:UA:3:Direct:20240131181336", "from_cluster": "DDSR.CMC", "mtime": "20240131181337.964", "parts": "1,123,1,0,0", "source": "WXO-DD", "sum": "d,6a78de9a730e81a87751f79f50c23294", "sundew_extension": "fromPxatx:CWAO:UA:3:Direct:20240131181336", "to_clusters": "DDSR.CMC,DDI.CMC,CMC,SCIENCE,EDM"}, "20240131181337.964 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___37295"]
["v02.post.bulletins.alphanumeric.20240131.UA.CWAO.18", {"atime": "20240131181413.110", "filename": "msg_ddsr-WXO-DD3_1a758da1110ac2344c49e3a7e9a6a828:fromPxatx:CWAO:UA:3:Direct:20240131181411", "from_cluster": "DDSR.CMC", "mtime": "20240131181413.110", "parts": "1,118,1,0,0", "source": "WXO-DD", "sum": "d,4829fb74f6f5575b97dfa5d058888dad", "sundew_extension": "fromPxatx:CWAO:UA:3:Direct:20240131181411", "to_clusters": "DDSR.CMC,DDI.CMC,CMC,SCIENCE,EDM"}, "20240131181413.110 https://dd4.weather.gc.ca /bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311814___47827"]

Changed the config to use a directory that does exist, and only accept the files I know are in the retry queue:

broker amqps://dd.weather.gc.ca/
topicPrefix v02.post

expire 1h

subtopic bulletins.alphanumeric.*.*.CWAO.#

directory /tmp/exists/
#accept .*UANT01_CWAO.*
accept .*UANT01_CWAO_311813___53188.*
accept .*UANT01_CWAO_311813___02502.*
accept .*UANT01_CWAO_311813___06373.*
accept .*UANT01_CWAO_311813___37295.*
accept .*UANT01_CWAO_311814___47827.*
reject .*

Log showing v2 uses the updated directory to successfully download:

$ sr_subscribe foreground test_change_dir
2024-01-31 18:24:38,477 [WARNING] extended option topicPrefix = ['v02.post'] (unknown or not declared)
2024-01-31 18:24:38,477 [INFO] sr_subscribe test_change_dir start
2024-01-31 18:24:38,478 [INFO] log settings start for sr_subscribe (version: 2.23.08):
2024-01-31 18:24:38,478 [INFO]  inflight=.tmp events=create|delete|link|modify use_pika=False topic_prefix=v02.post dry_run=False
2024-01-31 18:24:38,478 [INFO]  inline=False events=create|delete|link|modify use_amqplib=False topic_prefix=v02.post
2024-01-31 18:24:38,478 [INFO]  suppress_duplicates=False basis=path retry_mode=True retry_ttl=3600000ms tls_rigour=normal
2024-01-31 18:24:38,478 [INFO]  expire=3600000ms reset=False message_ttl=None prefetch=25 accept_unmatch=False delete=False poll_without_vip=True
2024-01-31 18:24:38,478 [INFO]  heartbeat=300 sanity_log_dead=450 default_mode=000 default_mode_dir=775 default_mode_log=600 discard=False durable=True
2024-01-31 18:24:38,478 [INFO]  declare_queue=True declare_exchange=True bind_queue=True
2024-01-31 18:24:38,478 [INFO]  post_on_start=True preserve_mode=True preserve_time=True realpath_post=False base_dir=None follow_symlinks=False
2024-01-31 18:24:38,478 [INFO]  mirror=False flatten=/ realpath_post=False strip=0 base_dir=None report_back=True log_reject=False
2024-01-31 18:24:38,478 [INFO]  Plugins configured:
2024-01-31 18:24:38,478 [INFO]          do_download:
2024-01-31 18:24:38,478 [INFO]          do_get     :
2024-01-31 18:24:38,478 [INFO]          on_message:
2024-01-31 18:24:38,478 [INFO]          on_data:
2024-01-31 18:24:38,478 [INFO]          on_part:
2024-01-31 18:24:38,478 [INFO]          on_file: File_Log
2024-01-31 18:24:38,478 [INFO]          on_post: Post_Log
2024-01-31 18:24:38,478 [INFO]          on_heartbeat: Hb_Log Hb_Memory RETRY
2024-01-31 18:24:38,478 [INFO]          on_report:
2024-01-31 18:24:38,478 [INFO]          on_start:
2024-01-31 18:24:38,478 [INFO]          on_stop:
2024-01-31 18:24:38,478 [INFO] log_settings end.
2024-01-31 18:24:38,478 [INFO] sr_subscribe run
2024-01-31 18:24:38,479 [INFO] AMQP  broker(dd.weather.gc.ca) user(anonymous) vhost(/)
2024-01-31 18:24:38,479 [INFO] Using amqp module (AMQP 0-9-1)
2024-01-31 18:24:38,501 [INFO] Binding queue q_anonymous.sr_subscribe.test_change_dir.75389274.42625677 with key v02.post.bulletins.alphanumeric.*.*.CWAO.# from exchange xpublic on broker amqps://anonymous@dd.weather.gc.ca/
2024-01-31 18:24:38,502 [INFO] declared queue q_anonymous.sr_subscribe.test_change_dir.75389274.42625677 (anonymous@dd.weather.gc.ca)
2024-01-31 18:24:38,504 [INFO] reading from to anonymous@dd.weather.gc.ca, exchange: xpublic
2024-01-31 18:24:38,506 [INFO] report_back to anonymous@dd.weather.gc.ca, exchange: xs_anonymous
2024-01-31 18:24:38,506 [INFO] sr_retry on_heartbeat
2024-01-31 18:24:38,512 [INFO] Number of messages in retry list 5
2024-01-31 18:24:38,513 [INFO] sr_retry on_heartbeat elapse 0.007136
2024-01-31 18:24:38,518 [INFO] Retrying notice  20240131181315.523 https://dd4.weather.gc.ca/bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___53188
2024-01-31 18:24:38,534 [INFO] file_log downloaded to: /tmp/exists/UANT01_CWAO_311813___53188
2024-01-31 18:24:38,536 [INFO] Retrying notice  20240131181328.723 https://dd4.weather.gc.ca/bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___02502
2024-01-31 18:24:38,552 [INFO] file_log downloaded to: /tmp/exists/UANT01_CWAO_311813___02502
2024-01-31 18:24:38,553 [INFO] Retrying notice  20240131181332.60 https://dd5.weather.gc.ca/bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___06373
2024-01-31 18:24:38,588 [INFO] file_log downloaded to: /tmp/exists/UANT01_CWAO_311813___06373
2024-01-31 18:24:38,589 [INFO] Retrying notice  20240131181337.964 https://dd4.weather.gc.ca/bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311813___37295
2024-01-31 18:24:38,603 [INFO] file_log downloaded to: /tmp/exists/UANT01_CWAO_311813___37295
2024-01-31 18:24:38,604 [INFO] Retrying notice  20240131181413.110 https://dd4.weather.gc.ca/bulletins/alphanumeric/20240131/UA/CWAO/18/UANT01_CWAO_311814___47827
2024-01-31 18:24:38,616 [INFO] file_log downloaded to: /tmp/exists/UANT01_CWAO_311814___47827
^C2024-01-31 18:24:44,794 [INFO] signal stop (SIGINT)
2024-01-31 18:24:44,794 [INFO] sr_subscribe stop
petersilva commented 8 months ago

fwiw: I tried the simple expedients of removing new_ fields in messags from the retry queue, but they result in crashes in the flow tests. 


Traceback (most recent call last):
  File "/home/peter/Sarracenia/sr3/sarracenia/instance.py", line 248, in <module>
    i.start()
  File "/home/peter/Sarracenia/sr3/sarracenia/instance.py", line 239, in start
    self.running_instance.run()
  File "/home/peter/Sarracenia/sr3/sarracenia/flow/__init__.py", line 529, in run
    if 'relPath' in m and m['new_file'] != m['relPath'].split('/')[-1]:
KeyError: 'new_file'
fractal%
reidsunderland commented 8 months ago

I'm a bit surprised by that, I thought it might work :(

petersilva commented 8 months ago

so there are two branches as PR's..

So the problem is that the new_ settings are generally set by updateFieldsAccepted based on the matched regex (accept clause) operant... and all the code is in the past when using after_accept() as the entry point to get stuff from the retry_queue.

1st Approach: #914 move retry get from after_accept to gather() entry point.

This also has the effect that if something in the configuration has changed to reject something in the retry queue, it will be rejected and not sent. so not just destination directory, but also accept/reject clauses get re-played in this scheme.

2nd Approach: 1st + remove all new_ and _deleteOnPost fields

I don't know which of the two approaches is better. There is a PR for both, and we should only merge 1.

This would be something to investigate for 3.00.52

reidsunderland commented 8 months ago

This also has the effect that if something in the configuration has changed to reject something in the retry queue, it will be rejected and not sent. so not just destination directory, but also accept/reject clauses get re-played in this scheme.

This is good! :) This is also how v2 works. We sometimes run into cases where downloads are failing because the source server is down, or sends are failing because the destination server is down. This causes the retry queue to build up with messages, and once the server is back online, sometimes we manually add rejects for old data that isn't needed anymore.

Is there any reason not to go with the 2nd approach? If the new_ and deleteOnPost fields are going to get deleted when restoring from the retry queue, I don't think there's any point in storing them in the first place?

petersilva commented 8 months ago

OK... I'm not sure if this applies... I'm just thinking about what could be a problem...

If we forget everything we did... then we will "pretend" we got the message only when we successfully transmit it.... so there will be no "delay" recorded in the message to reflect failed attempts to deliver. It will have the publication time, and the send time, and then people have to look in the logs to see records of failed attempts, rather than the message recording the fact.

Looking at the code... the only metric we record is "timeCompleted" a field only updated after successful transfer. So I think this worry is moot... there is nothing we record about reception time or retries in the message anyways. If we decide to add that... it would add a wrinkle to the fields to be removed... (an exception for those cases.)

A consequence of the above is that the messages are "received" the last time they are retried... so when we assign date/time (%Y%m%d and fiends) in delivery parameters, it will reflect time near to when a successful deliver occurred, rather than when it was received initially. Is that wrong? I don't know... just a worry.

Similarly... if there is a plugin that ... I dunno does something... (causes state change) then it will change again. For example, when retrying... it will be a duplicate of itself, and will be suppressed. so maybe we need a field "retry" that avoids duplicate suppression logic, but what if configuration changes are such that you want to change what a duplicate is, and now it does qualify as a duplicate? ... I don't know...

The sr3 design intent was to save the state of the message prior to delivery, so all the processing was re-used, and the original message processing was preserved. This issue is arguing against that design choice. So this is an argument for a design change, which is OK... it just means we think about it a bit more.

petersilva commented 8 months ago

the date processing thing... it means that the file may show up in different trees on different clients based on retries... xx.txt could show up in one hour tree on destination A, and a different one on B (one retry) and a third one on C (multiple retries.)

When data is queued for a long time (maintenance) all the dates of delivery will reflect the return to service... Again, I don't know if this is bad... It's just an effect to expect.

petersilva commented 8 months ago

Situation today when the problem @reidsunderland mentions occurs:

Obviously this is a lot less auto-magic that what @reidsunderland is asking for, but it only happens with the configuration changes. debate on relative frequence versus just plain re-starts becaus of outage.

A third cop-out way... if we can't decide... --reingest... have an option (command-line?) to have it use gather, but normally user after_accept.... options when we can't decide are sometimes a bad sign...

Also note that the log will have a complete record of retries... the "state" being lost is just within the message... not clear if that's important.

petersilva commented 8 months ago

yeah... I'm leaning more towards an option like --reingest... --retry_refilter ? --retry_gather? and defaulting to way sr3 does it today, with the option to behave the other way.... but then we need to get both ways to work... it's just an added routine (99% of the after_accept we had before we started.) with a guard to use one of gather() or after_accept to inject messages depending on the option.

suggestions about the name?

so in the case of a config error, on startup you supply ''--

petersilva commented 8 months ago
petersilva commented 8 months ago

Other thoughts

petersilva commented 8 months ago

Third PR available, this one adds a retry_refilter flag to switch between v2 and sr3 behaviours... except, I don't think nodupe is taken care of yet...

petersilva commented 8 months ago

ok the retry_refilter logic is merged, so default retry behaviour is as before, but retry_refilter option is available. However, the duplicate suppression logic will likely discard retried messages due to their having passed the duplicate suppression test before (guarateeed to be duplicates of themselves.) so likely need to re-introduce some form of isRetry from v2.