Add test for AM - Githubissues

andreleblanc11 commented 6 months ago

With more AM stuff being added in the source code, their should be a flow test dedicated to AM.

This test would:

Test pulling AM data from known sources (or from AM sender perhaps)
Test the AM renamer. Make sure it does its job correctly.
- We could confirm this by comparing operational data?
Test sr3 ingestors of AM data. Validate they are able to download.

andreleblanc11 commented 3 months ago

I was able to verify a proof of concept where the AM sender would get files from a watch/sarra pair then send to an AM receiver.

I think it would be best to utilize a similar flow for the flow test.

The flow watch -> (post locally) -> sarra -> (post locally) -> sender/am_send -> (AM socket) -> flow/amserver -> download locally

We already have a good variability of bulletins under samples/data/20200105/WXO-DD/bulletins/alphanumeric so it would be good to utilize these bulletins for the test.

I prefer having a local flow for the test because:

If we pull live data, the flow isn't big enough to cover most use cases. Having a local flow we can encounter every use case if we want (which we likely will).
It doesn't make sense to use AMTCP writers on the regionals for flow tests. We would need to run the tests from one machine only and we couldn't automate it through GitHub actions

petersilva commented 3 months ago

Sounds good, and you could even have the local download be the same directory that is watched, so it makes an infinite loop. Then in the log, duplicate suppression should stop it at some point.

andreleblanc11 commented 3 months ago

I've been able to get flow_setup.sh and flow_cleanup.sh to setup the flow and cleanup the flow effectively.

Next steps:

I don't think we need to have a flow_limit.sh in place. The test goes by and finishes in a couple of seconds.
Have flow_check.sh correctly verify the output of the file transfers, the messages sent/received, and possibly checksums?

petersilva commented 3 months ago

fwiw, I tried running the AM tests and they failed horribly... are they supposed to work for me?

andreleblanc11 commented 3 months ago

Yes they should be working.. that is interesting. What are the errors that are reported?

andreleblanc11 commented 3 months ago

I found a couple of problems.. Correcting them

andreleblanc11 commented 3 months ago

I now have the test passing on one node and not the other one and I have no idea why.

One will have the watch post with this topic. (the correct topic)

2024-06-07 19:02:12,622 [INFO] sarracenia.flowcb.log after_post posted to exchange: xs_tfeed_am topic: v03.post.bulletins_to_post.bulletins.20200105.UA.CWAO.02 a file with baseUrl: http://localhost:8001 relPath: bulletins_to_post/bulletins/20200105/UA/CWAO/02/UANT01_CWAO_050259___25058 rename: /bulletins_to_post/bulletins/20200105/UA/CWAO/02/UANT01_CWAO_050259___25058

And the other with this topic (the incorrect topic)

2024-06-07 19:34:44,427 [INFO] sarracenia.flowcb.log after_post posted to exchange: xs_tfeed_am topic: v03.post.20200105.UA.CWAO.02 a file with baseUrl: http://localhost:8001 relPath: 20200105/UA/CWAO/02/UANT01_CWAO_050259___25058 rename: /20200105/UA/CWAO/02/UANT01_CWAO_050259___25058

The configs are exactly the same and the sr_insects code is the same as well. sr3 version is also the same.

I even compared sr3 show watch/watch-dir and found nothing useful that was indicative of a problem.

MetPX / sr_insects

Add test for AM #37