search

JohnTheNerd / PySarra

A Python Sarracenia client without any of the bells-and-whistles.
The Unlicense
1 stars 0 forks source link

v03 heads up #4

Closed petersilva closed 4 years ago

petersilva commented 4 years ago

just to let you know, for the past year or so, we have been working on a new message payload format, as a result of limitations in the current ones and feed back from some international consultations. The current protocol version is identified by using the topic tree that starts with v02.post. Over the next year or two, we may migrate to v03.post. Differences in messages:

AMQP headers are no longer used to store key-value pairs. Instead, the message body is a JSON array. As a result, the formerly anonymous fields in the body of a v02 message are now key value pairs in the array: pubTime, baseUrl, and relPath.

for fields that are encoded, such as checksums, the encoding is changed to base64 (more compact representation)

https://github.com/MetPX/sarracenia/blob/master/doc/sr_postv3.7.rst

might still evolve slightly (new fields?) but we have done some important deployments of v03, and it is looking solid. no fire... nothing will be sprung on consumers suddenly, we haven´t looked at any migration strategy yet, but would not want to spring it on clients all of a sudden. Figured you would want to know far ahead of time.

I can supply some alternate data streams if you want a sample.

JohnTheNerd commented 4 years ago

is there a staging server of some sort where I could compare v03.post messages with what I currently get from Sarracenia? unless that's what you mean by the alternate data stream, of course

petersilva commented 4 years ago

v03 isn't available on the normal datamart stream. I have another experimental feed you can try just for format. We don't even have a plan for when main datamarts will move to v03. If you want to try it out (not the same products.)

if you connect to:

broker amqps://anonymous:anonymous@hpfx.collab.science.gc.ca exchange xs_pas037_wmosketch_public topic v03.post.#

and you will see a ton of messages in v03 format. This is just a toy data stream (selection from UNIDATA LDM feed.) to for interop experiments with some other countries.

JohnTheNerd commented 4 years ago

I have implemented support for v03.post while adding the test server you mentioned into a branch named v03-post and it seems to be functioning just fine.

as PySarra only uses a small subset of Sarracenia's functionality (it literally gets the URLs and throws away everything else), I can't see it breaking when v03.post is fully rolled out after which I'll just change the URL back to dd.weather.gc.ca, test to make sure everything works, and merge the branch into master.

thanks for the early heads-up! closing the issue for now, but please feel free to reopen or create another issue if you discover anything!