installing sumalgo plugin a failed: not found

[matthewdarwin]

What is the sumalgo plugin, why doesn't it work and why is it used for the new "metnote" data? I'm using this:

$ dpkg --list | grep metpx
ii  python3-metpx-sarracenia          2.17.02a1                      all          Meteorological Product Exchanger

Also if I run an md5sum on the file, I get a different value:

md5sum 20220724T150525.289Z_MSC_MetNotes.json
82bc0392141c3605dbd9e79d14cffe95  20220724T150525.289Z_MSC_MetNotes.json

error log:

2022-07-24 08:56:41,048 [INFO] sr_subscribe start
2022-07-24 08:56:41,048 [INFO] sr_subscribe run
2022-07-24 08:56:41,049 [INFO] AMQP  broker(hpfx.collab.science.gc.ca) user(anonymous) vhost(/)
2022-07-24 08:56:41,694 [INFO] Binding queue q_anonymous.sr_subscribe.dd7-hpfx.wx-dd.mar.davintech.ca with key v02.post.#.WXO-DD.metnotes.# from exchange xpublic on broker amqps://anonymous@hpfx.collab.science.gc.ca/
2022-07-24 08:56:42,040 [INFO] report_back to anonymous@hpfx.collab.science.gc.ca, exchange: xs_anonymous
2022-07-24 11:10:38,382 [ERROR] script not found a
2022-07-24 11:10:38,382 [ERROR] installing sumalgo plugin a failed: not found
2022-07-24 11:10:38,382 [ERROR] Type: <class 'FileNotFoundError'>, Value: [Errno 2] No such file or directory: 'a'
2022-07-24 11:10:38,382 [ERROR] for option sumalgo plugin a did not work
2022-07-24 11:10:38,382 [ERROR] sumalgo plugin a incorrect: does not set self.sumalgo
2022-07-24 11:10:38,382 [ERROR] sumflg a not working... set to 'd'
2022-07-24 11:10:38,385 [INFO] msg_log received: 20220724150525.387 https://hpfx.collab.science.gc.ca/20220724/WXO-DD/metnotes/20220724T150525.289Z_MSC_MetNotes.json topic=v02.post.20220724.WXO-DD.metnotes lag=312.999 sundew_extension=DMS:CMC:METNOTE:JSON: parts=1,34571,1,0,0 to_clusters=DDI,DDSR x-delay=0 sum=a,797d8221a35ef859e5e674b153a1125e source=MSC-DMS-OP from_cluster=DDSR.CMC
2022-07-24 11:10:38,386 [INFO] mjd_message sending: 797d8221a35ef859e5e674b153a1125e 34571 https://hpfx.collab.science.gc.ca/20220724/WXO-DD/metnotes/20220724T150525.289Z_MSC_MetNotes.json 20220724150525.387

[petersilva]

DMS does not use real checksums. We had to implement the "arbitrary" scheme.... where it's just a random number they picked. the sumalgo plugin a did not work means you are running an older version that does not support the "a" (arbitrary) checksum scheme.

[petersilva]

Met_Notes are a DMS product, as are SWOB... and CAP...

[petersilva]

in case you are asking... doesn't this make checksums pointless? yes... yes it does... wasn't my idea.

[petersilva]

2.17 is really old... there is 2.22 available... (the second digit is the two-digit year, so 2.17 is about five years old.)

[matthewdarwin]

Well there was no reason to upgrade before. And the announcement about this new data feed didn't say there was a specific version required.

Tried 2.22.06~ubuntu20.04.1, and AMQP won't connect.

AMQP cannot connect to hpfx.collab.science.gc.ca with [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

Need to figure out why.

[petersilva]

sigh... we are having problems with keeping our SSL certs uptodate... itś likely expired. for a day or two:

    tls_rigour lax

should let modern versions connect.

[matthewdarwin]

That option doesn't seem to do anything.

I changed /usr/lib/python3/dist-packages/amqp/transport.py line 440 to provide the O/S certificates file.

ca_certs='/etc/ssl/certs/ca-certificates.crt', do_handshake_on_connect=False,

Which seems to get us further. New error:

sr_amqp/build could not declare queue q_anonymous.sr_subscribe.dd4-hpfx.my-hostname-here (anonymous@hpfx.collab.science.gc.ca) with Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'q_anonymous.sr_subscribe.dd4-hpfx.my-hostname-here' in vhost '/': received 'true' but current is 'false'

[matthewdarwin]

I change the config file name, so now I get

sramqp/build could not declare queue q.sr_subscribe.dd4-hpfx3.my-hostname-here (anonymous@hpfx.collab.science.gc.ca) with Queue.declare: (403) ACCESSREFUSED - access to queue 'q.sr_subscribe.dd4-hpfx3.my-hostname-here in vhost '/' refused for user 'anonymous'

[petersilva]

in 2.17 the default durable setting was False ( queues are discarded when broker restarts) which we decided was bad, so the default was changed to True. This happenned in late 2018. If you have an old queue, they won't match. You can:

• set durable off in your configuration. you can run the subscriber now for a while, until it doesn't have any backlog.
• sr_subscribe cleanup your_config ... to remove the old queue definition (you lose queued data.)
• remove the durable off line.... so next time you start it will create a new queue with durable=True

Durable=True is better.

[petersilva]

I just checked the certificate for hpfx.collab.science.gc.ca and it looks fine. I don't know what your original problem was, it should be ok with just stock settings.

[matthewdarwin]

Thanks for all the help. I'm waiting for a metnote to be published so I can see if the data feed is working.

[matthewdarwin]

metnote download successful.