MetPX / sarracenia

https://MetPX.github.io/sarracenia
GNU General Public License v2.0
45 stars 22 forks source link

exchangeSplit option unclear and undocumented. #624

Open petersilva opened 1 year ago

petersilva commented 1 year ago

I don't remember what exchangeSplit does. I think the idea is that you have, say, a winnow with post_exchangeSplit making a bunch of exchanges and sending to them with sorting based on modulo of check sums. the complementary option would be something where you declare a subscriber, where each instance subscribes to a corresponding exchange

in winnow config:


winnow
    post_exchange xs_foo
    post_exchange_split 4

  ---> declares exchanges  xs_foo00, xs_foo01, xs_foo02, xs_foo03

in subscriber config

  instances 4
   exchange xs_foo
   exchangeSplit yes

  subtopic A

should result in different bindings for each instance:

instance 1 should bind it's queue to topic A of exchange xs_foo00

instance 2 should bind its queue to topic A of exchange xs_foo01

instance 3 should bind it's queue to topic A of exchange xs_foo02

...

So... I think that is what it does and is for. It makes it easier to declare subscribers to a queue.

Looking at the repository:

The goal of this issue is to test it out, and make a decision about whether to

a) finish it, and write some documentation, or b) remove it.

reidsunderland commented 5 months ago

I tried to use exchangeSplit:

1) exchangeSplit is defined in both count_options and a flag_options in config.py. sr3 show shows True/False, so it's treating it as a flag option.

If we keep it as a flag option, then the user needs to define the number of instances. 

2) instances > 1 have issues with writing/setting/reading the qname:

    2024-05-13 20:24:18,972 [CRITICAL] 1600448 sarracenia.config _resolveQueueName failed to read queue name from /net/local/home/sunderlandr/.cache/sr3/subscribe/configname/subscribe.configname.bunnymaster02.qname
The `subscribe.configname.bunnymaster01.qname` gets created, but the others don't.

3) It doesn't seem like the exchange name is getting set correctly:

```
# From instance 1, the exchange in the config is xs_TESTEXCH_, so it should be using xs_TESTEXCH_00

2024-05-13 20:24:20,885 [ERROR] sarracenia.moth.amqp getSetup AMQP getSetup failed to localhost with Queue.bind: (404)     NOT_FOUND - no exchange 'xs_TESTEXCH_' in vhost '/'
```

sr3 show does show all the exchanges though: 

```
 'exchange': ['xs_TESTEXCH_00', 'xs_TESTEXCH_01', 'xs_TESTEXCH_02',
              'xs_TESTEXCH_03', 'xs_TESTEXCH_04'],
```
petersilva commented 5 months ago

Right... so if each instance is subscribing to e different exchange, then each instance needs a different queue. so we need to add an instance number to the queue_name somehow. then each instance's binding swill be between one queue and one exchange.

petersilva commented 5 months ago

Is it creating multiple queues? or maybe that's the part that is missing?