Add high priority message channel for each actor

Remove Recipient::remaining_capacity()

Would have to accept extra parameter once we add priority channel. That would be well possible, but perhaps this method in not really needed once we have prio channels? Let's try without it first.

Add high priority message channel for each actor

Actor trait has gained new optional method priority(message: &M) which determines priority of each message before being sent.

Addr grows by one extra flume::Receiver. Recipient doesn't grow as its message_tx payload is boxed.

Support specifying capacity independently for normal and high-priority channel

When porting portal to use priority channels, I've realized actor systems may want to set capacity of the normal channel as low as 1 (for actors that cannot keep up with the pace, but for which that doesn't matter).

Producers of their "bulk" low-priority messages will usually ignore their channels being full. OTOH producers of their important high-priority messages may like to fail when their priority channel is full (as ignoring that may have dire consequences). It would be thus impractical if the priority channel got similarly small capacity. (actor needs to finish at least one event loop to get to process the high-prio messages).

Forward priority to inner actor in wrapping actors

In the Timed wrapper this is non-trivial, so some extra explanation docs are added.

Fixes #22.

With this change, are we maintaining roughly the same performance we gained when switching to flume in the prerequisite PR?

I looks like there is a slight regression compared to right after porting to flume:

circular/circular (2 actors)                                                                             
                        time:   [3.1650 ms 3.1949 ms 3.2268 ms]
                        thrpt:  [309.90 Kelem/s 312.99 Kelem/s 315.96 Kelem/s]
                 change:
                        time:   [+6.1280% +7.4680% +8.7130%] (p = 0.00 < 0.05)
                        thrpt:  [-8.0147% -6.9490% -5.7741%]
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
Benchmarking circular/circular (native CPU count - 1): Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 10.0s. You may wish to increase target time to 18.8s, enable flat sampling, or reduce sample count to 50.
circular/circular (native CPU count - 1)                                                                             
                        time:   [4.1088 ms 4.1343 ms 4.1591 ms]
                        thrpt:  [240.44 Kelem/s 241.88 Kelem/s 243.38 Kelem/s]
                 change:
                        time:   [+7.9763% +10.226% +12.621%] (p = 0.00 < 0.05)
                        thrpt:  [-11.207% -9.2773% -7.3871%]
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high mild
circular/circular (native CPU count)                                                                            
                        time:   [22.316 ms 23.207 ms 24.119 ms]
                        thrpt:  [41.460 Kelem/s 43.090 Kelem/s 44.811 Kelem/s]
                 change:
                        time:   [+1.5387% +6.6061% +12.139%] (p = 0.02 < 0.05)
                        thrpt:  [-10.825% -6.1967% -1.5154%]
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
circular/circular (50 actors)                                                                            
                        time:   [19.662 ms 20.622 ms 21.649 ms]
                        thrpt:  [46.191 Kelem/s 48.492 Kelem/s 50.859 Kelem/s]
                 change:
                        time:   [-11.337% -4.4279% +2.8930%] (p = 0.25 > 0.05)
                        thrpt:  [-2.8117% +4.6330% +12.787%]
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

I guess the reason is the select loop now has to check the priority channel first, i.e. one extra operation per message received. The regression of adding the channel is fortunately lower than what we gained by using flume in the first place. I have to say that the benches are rather fluctuating at my laptop (in one run I got 51.5 Kelem/s ~ +17% for circular/circular (native CPU count)), so we should take them with a grain of salt.

I have pushed one extra commit, to also update wrapping actors.[1] With that it should be hopefully complete.

[1] This illustrates one reason I don't usually like trait methods with default implementations: it is too easy to forget implementing them, especially if they are added later. OTOH I didn't like not adding the default implementation either: having the default allows actor system not care at all about the priority if they don't want to.

tonarino / actor