Allow extensibility by freeing Broker internals from scope jail

[scottp-dpaw]

My company is retrofitting one of our Rascal-using apps to have dynamic queuing support; basically for publisher processes to assert a target-specific queue at runtime and send to a topic exchange with a routing key, and for subscriber processes to monitor multiple queues and create new subscriptions in the event of new queues being created.

Now, I understand that all of this is outside of the remit of Rascal; it's a high-level layer for a specific queuing style. Under normal circumstances, we would make a local subclass that shims just the bits we need to get the functionality we require. However in Rascal we can't do this, as the classes are function closures with attributes grafted on. Furthermore, a lot of "private" state variables only exist in the function scope, making it impossible to access from outside of the Rascal library.

As such, we've had to make an internal fork of Rascal which is exactly the same only with some state variables exposed (see https://github.com/scottp-dpaw/rascal/commit/4c83018e8a4a398e254bee68f375692388eb6e8e).

The question I have: would Rascal consider moving these private variables out of the scope for extensibility purposes? Obviously they wouldn't be part of the supported API (maybe prefixed with _), and users would be on the hook for keeping the shims working across library upgrades, but it would save the headache of having to fork the entire library.

[cressie176]

Hi @scottp-dpaw,

Thank you for your suggestion and link to the fork. I'm sorry, but I wouldn't consider making these variables externally accessible because it would break encapsulation. This is what ensures state and behaviour are kept together and in a single place, which is key for maintaining flexibility and minimising bugs.

The requirement to dynamically add queues, exchanges, publications and subscriptions comes up from time to time. I can see it would be useful, and happy to discuss approaches. The things I can predict causing problems are

1. Rascal has a defaulting / augmentation / configuration / validation phase which currently works on the entire config. Adding components (queues, exchanges, etc) individually will necessitate refactoring and reusing this.

2. Rascal would have no knowledge of the dynamically created components, so they would not automatically survive restarts. I don't this is Rascal's responsibility to solve however. I'm curious to know how you're coping with this?

[scottp-dpaw]

Thanks for the quick reply. I figured this would be the case, but couldn't hurt to ask.

Re. 2: not sure if there's anything I'm missing, but our mods update the internal Broker state as we go.

• Every time a dynamic queue is initialised:

  • Call broker.vhosts[vhost_id].borrowChannel()
  • Use the channel to assert then check the exchange configuration
  • Use the channel to assert then check the queue configuration
  • Use the channel to assert then check the queue binding to the exchange
  • Call broker.vhosts[vhost_id].returnChannel(channel)
  • Create a Subscription config for this queue, same as what would need to be specified in the normal pipeline
  • If a matching name isn't in broker.subscriptions
  • Send the config through Subscription.create() and get an object
  • Add the object with broker._addSubscription()
  • Create a Publication config for this queue, same as what would need to be specified in the normal pipeline
  • If a matching name isn't in broker.publications
  • Send the config through Publication.create() and get an object
  • Add the object with broker._addPublication()

• For subscription workers:

  • On init:
    • Run the normal config-based pipeline on init to get a BrokerAsPromised (excluding dynamic queue info)
    • Use the RabbitMQ management interface to get a list of queue bindings for the exchange
    • Get the Broker from inside the BrokerAsPromised
    • Add dynamic queues for all bindings (see above)
    • Add a subscription with broker.subscribe() for all of the dynamic queues to process messages
    • Add a subscription to a special event exchange queue which fires on new bindings; use this to trigger adding a new dynamic queue

• For publication workers:

  • On init:
  • Run the normal config-based pipeline on init to get a BrokerAsPromised (excluding dynamic queue info)
  • On receiving a request:
  • Get the Broker from inside the BrokerAsPromised
  • Add a dynamic queue based on the request target (see above)
  • Publish message to the exchange with broker.publish()

[cressie176]

Thanks for the info @scottp-dpaw. I'm going to close this for now, but will consider adding the ability to add queues/topics/consumers/producers dynamically in a future release