2022-05-31 meeting agenda

[MichalisPanayides]

What has been done:

• Gregynog ✔️
• Booking flights/hotels/registrations for conferences ⌚
• Julia course ✔️
• Fixed Reinforcement Learning algorithm bug ✔️
• Output of RL algorithm 💯🎉

To be discussed:

• [x] Problem with simulation - between state probabilities matrix and server rates matrix (these are not ciw issues)
  • ciw state tracker is updated AFTER getting the service time     \(@^0^@)/
  • ciw releases blocked individuals AFTER getting the service time    (っ °Д °;)っ
• [x] Notebook of results form RL algorithm     (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧

To be done:

• Simulated annealing of RL algorithm 🔢
• Thesis 😭
• Finish up all talks to be given 😵‍💫
• Other things I want to do:
  • Direct formula for blocking time ➡🚫
  • Closed form formula for steady states probabilities ☢
    
    
    
    
    
    

           All hail the holy meeting agenda

( ﾉ ﾟｰﾟ)ﾉ( ﾉ ﾟｰﾟ)ﾉ( ﾉ ﾟｰﾟ)ﾉ( ﾉ ﾟｰﾟ)ﾉ     ＼(ﾟｰﾟ＼)＼(ﾟｰﾟ＼)＼(ﾟｰﾟ＼)＼(ﾟｰﾟ＼)

[MichalisPanayides]

Two problems with state probabilities matrix and server rates matrix

Original thinking:

Markov chain set of states

State probabilities array:

Server rates matrix

I originally thought that the set of states from the simulation correspond to the server rates exactly. Meaning that I could just look at the current state of the simulation and use the rate from the equivalent position of the server rates matrix.

That is not the case because of two major problems.

Note: Need to think of when a service actually starts

Problem 1 - ciw state tracker is updated AFTER getting the service time

When we think about this for the first couple of states it makes sense (kinda). e.g Consider an example where C = 2

• System at state (0, 0) -> Arrival -> Service with rate μ_(0,0) -> System at state (0,1)
• System at state (0, 1) -> Arrival -> Service with rate μ_(0,1) -> System at state (0,2)
• System at state (0, 2) -> Arrival -> No service since system is full -> System at state (0,3)
• System at state (0, 3) -> Service ends -> Service with rate μ_(0,3) -> System at state (0,2) . . .

Solution: Instead of using ciw's state tracker I am now looking at the number of individuals at each node manually to get the actual state.

--------------------------------------Problem solved--------------------------------------

Problem 2 - ciw releases blocked individuals AFTER getting the service time

At our Markov model new individuals are going in service AT THE SAME TIME as releasing blocked individuals. In ciw it's also at the same time but the service computation is written before releasing blocked individuals. This is problematic for states on the threshold column. For a brief moment the system goes to potentially invalid states (i.e. the column of states before the threshold column).

Solution: Where we get the service time, check if we are at those invalid states and force the created distribution to sample from the correct ones.

[drvinceknight]

This all sounds excellent @11michalis11.

Regarding problem 1, I believe it's possible to implement and use a bespoke state tracker in ciw (you can pass a bespoke one to the simulation: https://ciw.readthedocs.io/en/latest/Reference/state_trackers.html?highlight=state%20tracker). Is this what you're doing or are you doing, and if not would that be a good idea to do?

[MichalisPanayides]

This all sounds excellent @11michalis11.

Regarding problem 1, I believe it's possible to implement and use a bespoke state tracker in ciw (you can pass a bespoke one to the simulation: https://ciw.readthedocs.io/en/latest/Reference/state_trackers.html?highlight=state%20tracker). Is this what you're doing or are you doing, and if not would that be a good idea to do?

I think currently all the state tracker is doing, is it collects the probability at each state, to be viewed at the end of the simulation. What I want to do is look at the current state of the simulation while running the simulation, which I was able to do without using the state tracker.

The reason why I couldn't use the state tracker (or any other custom tracker) is because ciw runs first self.accept_individual and then self.update_statetracker. That means the service time is sampled before updating the state tracker. I want to sample my service time ($\mu$) from the current state.

If that was confusing I can explain it better in our meeting. Overall this was a fun problem to solve though😄

[drvinceknight]

This makes sense. Thanks.

[MichalisPanayides]

This week's TODO:

• [x] Email to Julian Scheur for annual review meeting - 21st of June 13:00
  • Alternatives: 21/06 0900-1000, 22/06 after 13000, 24/06 1400-1500
• Reinforcement Learning plots:
  • [x] Run RL without the upper boundary
  • [ ] Plot all servers rates using with different starting μ's
  • [ ] Get a 3rd plot that's the weighted mean ratio of the rate to the max rate
    • Ratio = (rate at state / max possible rate)
    • i.e. Plot = ratio * pi_(u, v) over all iterations
  • [ ] Utility function 8 = e (Proportion of lost inds) + (1 - e) (Service time)
• [x] CORS/INFORMS talk:
  • Run RL with flooded model (miniature of Covid)