How to use picoquic multipath APIs

[abc1590011]

I have 2 network cards，maybe more. If I create 1 stream and then I want to send data on it using 2 network cards, how should I use picoquic multipath APIs ? If I create 2 streams, and then I want to use 2 network cards to send data on the 2 streams respectively, how should I use the picoquic multipath APIs ?

[hfstco]

Hey abc,

Before you establish your QUIC connection, you should prepare two things.

First, you should enable path callback events. This could be easily done by calling: picoquic_enable_path_callbacks_default(quic, 1); Now your callback function will be called with the _picoquic_callback_pathavailable, _picoquic_callback_pathsuspended, _picoquic_callback_pathdeleted and _picoquic_callback_path_qualitychanged events.

You have to enable multipath too: picoquic_set_default_multipath_option(quic, option); There are two multipath options to choose from. I use the first option. (1, QUIC multipath draft) There is another multipath option "simple multipath" too. (2, https://github.com/huitema/quicmpath)

Now you establish the QUIC connection with your "primary" interface. When the QUIC connection is established (maybe before?), you can probe a new path. 

// probe a new path (SAT)
struct sockaddr_storage addr_from;
int addr_from_is_name = 0;
struct sockaddr_storage addr_to;
int addr_to_is_name = 0;

picoquic_get_server_address("172.30.21.3", 6000, &addr_from, &addr_from_is_name); // remote addr
picoquic_get_server_address("172.30.20.2", 0, &addr_to, &addr_to_is_name); // local addr

picoquic_probe_new_path_ex(quic_ctx, (struct sockaddr *) &addr_from, (struct sockaddr *) &addr_to, 0, picoquic_current_time(), 0);

After some time (validating your path), the _picoquic_callback_pathavailable event should be raised through the callback function with the new _unique_pathid (stream_id parameter). It tells you that your new path is established and available.

Now you can set a specific path affinity for your streams with picoquic_set_stream_path_affinity(quic_ctx, stream_id, unique_path_id); or let picoquic decide which path to choose.

Best regards, Matthias

[abc1590011]

@hfstco Thank you very much for your answer.

Recently I have been reading the code for path choose in picoquic, I found that it probably in a polling way to select the path.

But that's not the case，I have 2 networks card in the client, send stream data 50 times. In the server side, I can see the 2 networks card are working, but not in a polling way, and more importantly, the number of interactions is wrong.

In my understanding, I send stream data 50 times in one stream id, network card A send 25 times, network card B send 25 times. However, the actual situation is network card A send 50 times, network card B send 50 times, They may not be sending streaming data, But shouldn't interact so often. In this case, network card aggregation is meaningless.

Can you tell me why, look forward to your letter. Thanks again.

[huitema]

@hfstco I am not sure that I understand your measurements. The picoquic network layer will call the "prepare to send" API when the stack says it is ready to send something, but the this is not a precise evaluation. Instead of keeping a lot of state to know whether there really is something to send, the stack will try to build a packet, and if comes empty it will not send anything. If you measure an intermediate point in the stack, the measurements may be misleading.

The tests check that data is sent over several paths, but it is rare to achieve a perfect 50%/50% split. The scheduling depend for example on the state of the congestion control, which will often be different for different paths. For example, the path that started first will reach full bandwidth usage before the other path.

There may also be an issue of sending control frames, such as acknowledgements. Maybe that could be improved. Do you have a network trace showing the packets and frames sent on each path, e.g., a qlog trace?

[abc1590011]

@huitema Thank you very much for your answer. I did no use qlog, just print recvmsg accepted data. I found that the server did receive data from the client's 2 network cards.

But before the server receives the client stream data, it also receives some other data from the client's 2 network cards that I don't know about. I'm confused about that.

Can you show me how to use qlog to get the decrypted data for each path? Because I use wireshark, I can't decrypt the data of both paths.

look forward to your letter. Thanks again.

[huitema]

The packets of length 55 are most probably acknowledgements. There was a bug in the production of acknowledgements, sending way too many acks in some conditions. The bug was fixed in PR #1545, which was merged three weeks ago. I think that you are using an older version -- I see the bad logging format at sender.d::1570, and that too was fixed in PR #1545. Updating to the latest version should improve your measurements.

[abc1590011]

@huitema Thanks for your reminding. I pulled the lastest master branch, but that's still the case.

Can you show me how to use qlog to get the decrypted data for each path? Because I use wireshark, I can't decrypt the data of both paths. look forward to your letter. Thanks again.

[huitema]

This is explained in the file doc/QLOG.md

[SupradhaBhat]

@hfstco Thank you very much for your answer.

Recently I have been reading the code for path choose in picoquic, I found that it probably in a polling way to select the path.

But that's not the case，I have 2 networks card in the client, send stream data 50 times. In the server side, I can see the 2 networks card are working, but not in a polling way, and more importantly, the number of interactions is wrong.

In my understanding, I send stream data 50 times in one stream id, network card A send 25 times, network card B send 25 times. However, the actual situation is network card A send 50 times, network card B send 50 times, They may not be sending streaming data, But shouldn't interact so often. In this case, network card aggregation is meaningless.

Can you tell me why, look forward to your letter. Thanks again.

I notice that you are mentioning that you were able to use two interfaces of your system, though with some issues in packet scheduling. I've been working on using multiple interfaces using Picoquic

I am currently trying with the command : ./picoquicdemo -M -A 192.168.0.0,192.168.92.1 -o ./client_files localhost 4433 doc.pdf

I am doubtful if my actual physical interfaces are being used, because only 1 packet is being sent on paths other than path[0]

Could you please share the command or configuration you used to enable multiple physical interfaces? It would be really helpful for me to understand how you managed to get both interfaces working.

[huitema]

You are connecting to localhost. No packet will leave your computer, it is all internal.