Sample code for file transfer

[basu13]

Can you provide any sample code for client and server file transfer in c++/c using picoquic. I am successfully able to build it but I don't how to use it. Successfully executed those commands that @huitema told in previous comments picoquicdemo -p 4433 for server and picoquicdemo -D ::1 4433 /200000000. It gives speed about 514mbps for me. Please help me for file transfer.

[huitema]

Yes, there is an issue. The picoquicdemo code is of course more a test tool than a sample. It does have support for file transfer, albeit in a limited way, with 2 command line options, one for the server and one for the client:

  -o folder             Folder where client writes downloaded files,
                        defaults to current directory.
  -w folder             Folder containing web pages served by server

As for sample application, you can look at the quicdoq project, a simple implementation of DNS over QUIC.

Or, if you have specific requirements for a file transfer service, we could have that as a case study.

[basu13]

Thank you @huitema I want to develop p2p project for file transfer and also want to use QUIC protocol instead of tcp/ip to enhance file transfer speed. Is it possible?

[huitema]

That's an interesting project indeed. First step would be to write out how you want your application to operate, how peers find each other, how file transfer is supposed to happen, etc. What application level protocol will you use, HTTP-3 or something like BitTorrent, including features like splitting files in chunks and tit-for-tat selection of servers? What are your congestion control requirements, and do you need to support something like LEDBAT? What about traversal of NAT and firewalls?

Once we have a spec in some shape, we can start looking at implementations. With Picoquic, that mostly entails writing the application protocol specific callback method.

[huitema]

@basu13 I could design a small file transfer application as a "coding sample", but I wonder what kind of specification would meet your requirement. Possible list:

1) Something similar to curl: get a file from a specified URL. Use HTTP3 server. Just write the client. But then, supporting all curl options might be a bit large.

2) Something similar to scp. Get or put a file or a set of files from/to a remote host. The hard part there is client authentication on the server side.

3) Something similar to anonymous FTP, only supporting reading files and listing directories.

Any preference?

[basu13]

Thanks @huitema first my target is to implement file transfer on local network LAN. Just server side will open the socket and client side will join using IP and port then client side will send which file it require then Server side will acknowledge that file is available and will send the file. Similar like peer to peer file transfer.

[huitema]

@basu13 I see. Yes, that would make a simple code sample. I am a tiny bit concerned with security and discovery issues, but those can be worked on later.

[basu13]

@huitema If we have some base model then we can enhance its security too.

[huitema]

OK, I will see how to set up a lean code sample with a simple file transfer. How about:

1) App protocol defined as "one stream per file": client opens the stream, sends a message containing the name of the requested file. Server sends "reset stream" if file does not exist, file data in pure binary form if file is present. End of file is marked by end of stream.

2) Server is started with port number & specification of a directory from where to read files.

3) Client is started with server IP and port number, specification of a directory where to receive files, and list of files to request. Stdout provides a receive report, as in file_xxx: Success or file_xxx: Error NNN

[basu13]

Yes thats what i am looking for :) Thanks

[huitema]

@basu13 Sorry for the delay, I was rather busy last week. But I finally prepared the simple file sample that you requested. Please take a look at PR #965.

[codianselme]

@ basu13 Désolé pour le retard, j'étais plutôt occupé la semaine dernière. Mais j'ai finalement préparé l'exemple de fichier simple que vous avez demandé. Veuillez consulter le PR # 965 .

Usage: ./picoquic_sample client server_name port folder *queried_file or : ./picoquic_sample server port cert_file private_key_file folder

I would like to know what the following arguments refer to cert_file, private_key_file folder and folder * queried_file. I wanted to test the code but I did not understand the use of these arguments

[huitema]

@cadianselme: please check the text that was added as sample/README.md by @tbarbette. I think that it answers your questions.

[codianselme]

With what command can I compile the sample.c code using GCC? I used this one gcc sample.c -L ../ libpicoquic-core.a -I ../ picoquic / but I have a feedback that I don't understand well

[huitema]

Using cmake on the solution will create a make file. "make picoquic_sample" should work. Otherwise, you need to create your own make file.

[codianselme]

How can I use the server-client source code quic in the sample folder outside the picoquic project folder ?

[huitema]

@codianselme OK, I see your point. You can see from the file CMakeList.txt that the sample has a set of dependencies:

add_executable(picoquic_sample
    sample/sample.c
    sample/sample_client.c
    sample/sample_server.c
)

target_link_libraries(picoquic_sample
    picoquic-log
    picoquic-core
    ${PTLS_LIBRARIES}
    ${OPENSSL_LIBRARIES}
    ${CMAKE_DL_LIBS}
    ${CMAKE_THREAD_LIBS_INIT}
)

target_include_directories(picoquic_sample PRIVATE loglib picoquic)

If you write a similar file for just the sample, the libraries picoquic-log and picoquic-core will have to be external, in much the same way that the ${PTLS_LIBRARIES} are handled in the current file.

Of course, I am personally very familiar with the structure of the code, and things seem easy to me that are not so easy for you. Let's try to resolve your issues here, and then to sum up the results and add them to the documentation.

[codianselme]

Ok, then what advice do you recommend while waiting? should i wait for the documentation update?

[huitema]

Can you take a look at the project quicdoq ? It is an example of project built using picoquic. You could copy and adapt the CMakeList.txt file of that project, replacing references to the quicdoq code by references to your port of the sample, then use cmake to create a make file for your sample.

[codianselme]

Okay I will try. thank you

[huitema]

@codianselme I added some information to the sample's README. Does it help?

[codianselme]

@codianselme I added some information to the sample's README. Does it help?

I try to explore it

[codianselme]

I created a folder and put all the files mentioned in the information added to the sample README file, then I used the following command

gcc sample_client.c -L ../ libpicoquic-core.a -I.

but i have this /tmp/cclBQnu3.o: In the "main" function: sample.c :(. text + 0x1b1): undefined reference to "picoquic_sample_client" sample.c :(. text + 0x239): undefined reference to "picoquic_sample_server" collect2: error: ld returned 1 exit status

[huitema]

When building the main program, you need to link sample.c with sample_client.o and sample_server.o, as well as with the required libraries.

From what you are telling me, you don't find that natural. I suppose that I could eliminate "sample.c", and make the client and server self contained, so that "gcc sample_client.c" would just work.

You will still need to link with other libraries, not just "libpicoquic-core.a" but also the picotls libraries. Changing that would require a lot more work.

[codianselme]

Can you take a look at the project quicdoq ? It is an example of project built using picoquic. You could copy and adapt the CMakeList.txt file of that project, replacing references to the quicdoq code by references to your port of the sample, then use cmake to create a make file for your sample.

I have taken a look at the quicdoq project but I cannot compile it to make it work. I have this:

➜ quicdoq git:(master) ✗ make [ 12%] Building C object CMakeFiles/quicdoq-core.dir/quicdoq/quicdoq.c.o /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c: In function ‘quicdoq_callback_data’: /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:105:42: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Cannot create server context for server stream #%llu.\n", (unsigned long long)stream_id); ^~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:105:70: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Cannot create server context for server stream #%llu.\n", (unsigned long long)stream_id); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:114:46: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Cannot create query context for server stream #%llu\n", (unsigned long long)stream_id); ^~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:114:74: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Cannot create query context for server stream #%llu\n", (unsigned long long)stream_id); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:134:42: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(stream_ctx->query_ctx->quic, &stream_ctx->query_ctx->cid, "Quicdoq: Incoming query too long for server stream #%llu.\n", (unsigned long long)stream_id); ^~~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:134:71: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(stream_ctx->query_ctx->quic, &stream_ctx->query_ctx->cid, "Quicdoq: Incoming query too long for server stream #%llu.\n", (unsigned long long)stream_id); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:161:38: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Data arrived on client stream #%llu before context creation.\n", (unsigned long long)stream_id); ^~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:161:66: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(picoquic_get_quic_ctx(cnx), &cid, "Quicdoq: Data arrived on client stream #%llu before context creation.\n", (unsigned long long)stream_id); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:167:42: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(stream_ctx->query_ctx->quic, &stream_ctx->query_ctx->cid, "Quicdoq: Incoming response too long for client stream #%llu.\n", (unsigned long long)stream_id); ^~~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:167:71: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(stream_ctx->query_ctx->quic, &stream_ctx->query_ctx->cid, "Quicdoq: Incoming response too long for client stream #%llu.\n", (unsigned long long)stream_id); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c: In function ‘quicdoq_create_client_cnx’: /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:364:42: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(quicdoq_ctx->quic, &cid, "Quicdoq: Could not start the connection to %s.\n", (sni == NULL)?"":sni); ^~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:364:61: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(quicdoq_ctx->quic, &cid, "Quicdoq: Could not start the connection to %s.\n", (sni == NULL)?"":sni); ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c: In function ‘quicdoq_post_response’: /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:695:30: error: passing argument 1 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(query_ctx->quic, &query_ctx->cid, "Response #%d received at cnx time: %"PRIu64 "us.\n", query_ctx->query_id, ^~~~~ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘picoquic_cnx_t {aka struct st_picoquic_cnx_t }’ but argument is of type ‘picoquic_quic_t {aka struct st_picoquic_quic_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char fmt, ...); ^~~~~~~~ /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:695:47: error: passing argument 2 of ‘picoquic_log_app_message’ from incompatible pointer type [-Werror=incompatible-pointer-types] picoquic_log_app_message(query_ctx->quic, &query_ctx->cid, "Response #%d received at cnx time: %"PRIu64 "us.\n", query_ctx->query_id, ^ In file included from /home/anselme/Bureau/Labo/quicdoq/quicdoq/quicdoq.c:26:0: /home/anselme/Bureau/Labo/picoquic/picoquic/picoquic.h:364:6: note: expected ‘const char ’ but argument is of type ‘picoquic_connection_id_t {aka struct st_picoquic_connection_id_t }’ void picoquic_log_app_message(picoquic_cnx_t cnx, const char* fmt, ...); ^~~~~~~~ cc1: all warnings being treated as errors CMakeFiles/quicdoq-core.dir/build.make:62: recipe for target 'CMakeFiles/quicdoq-core.dir/quicdoq/quicdoq.c.o' failed make[2]: [CMakeFiles/quicdoq-core.dir/quicdoq/quicdoq.c.o] Error 1 CMakeFiles/Makefile2:99: recipe for target 'CMakeFiles/quicdoq-core.dir/all' failed make[1]: [CMakeFiles/quicdoq-core.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: *** [all] Error 2

I tried to resolve the error but couldn't

[huitema]

My fault, sorry. A change in the picoquic implementation was not applied to quicdoq. Will fix.

[codianselme]

OK, thanks

[huitema]

@codianselme Quicdoq issue is fixed now.

[codianselme]

Ok good,

Say, is it normal that I have this with picoquic?

picoquic git: (master) ✗ ./picoquic_sample server 4433 ./ca-cert.pem ./server-key.pem ./server_files Enter PEM pass phrase: Server ready on port 4433 [1] 22682 segmentation fault (core dumped) ./picoquic_sample server 4433 ./ca-cert.pem ./server-key.pem ./server_files

➜ picoquic git: (master) ✗ ./picoquic_sample client anselme 4433 server_files index.htm page.html test.txt No token file present. Will create one as . Starting connection to anselme, port 4433 Initial connection ID: ded4e2dce8389a97 Opened stream 0 for file index.htm Opened stream 4 for file page.html Opened stream 8 for file test.txt Connection closed. [1] 22683 segmentation fault (core dumped) ./picoquic_sample client anselme 4433 server_files index.htm page.html

[huitema]

Seg faults are never normal! Do you have a stack trace showing where that happened?

Looking at you data, the only weird part is "server_files" in:

./picoquic_sample client anselme 4433 server_files index.htm page.html

That means you are asking the client to copy incoming files in the directory from which the server is reading them. For example, the server would open "page.html" for reading, and the client would try to open the same file at the same time for writing. That cannot work.

On the other hand, that should cause an error message, not a crash.

[codianselme]

In this case, which command should I use to copy data from the server?

[codianselme]

Regarding quicdoq, can I have an example use case? A bit like with Use: ./picoquic_sample client server_name port folder * queried_file gold : ./picoquic_sample server port cert_file private_key_file folder

[huitema]

To copy data from the server, you should create a "client directory" separate from the server directory. Then do something like:

./picoquic_sample client anselme 4433 client_files index.htm page.html test.txt

[huitema]

Actually, checking the code, I realize that I made a shortcut in the sample. I assume that there is a trailing "/" after the directory name. So calling the server should be:

./picoquic_sample server 4433 cert_file private_key_file server_files/

And the client:

./picoquic_sample client anselme 4433 client_files/ index.htm page.html

[huitema]

@codianselme I have fixed the trailing "/" issue. I tried to reproduce your crash on both Windows and Ubuntu, but could not. I am not sure that exchanging messages in this Github issue is the most efficient way to progress. Do you want to have a short conversation using a chat room or a conference call? If that's a good idea, send me your coordinates by email to "huitema@huitema.net"; I can use any of Signal, Whatsapp or Telegram.

[codianselme]

Ok, but long before that you planned a command to start the server and the client with Quicdoq?

[huitema]

Sorry, I forgot briefly about Quicdoq. The behavior of client and server is explained at https://github.com/private-octopus/quicdoq#running-the-demo-application.

[huitema]

Closing this issue for now, will reopen if there is specific feedback.

[jeromelau1984]

Dear huitema: How do I use the sample to send and receive data in both directions？ I can understand the process of the client downloading data from the server, but on the client side, there was only one picoquic_callback_prepare_to_send callback event before picoquic_file_open, What can I do to send user data from client to server.

[huitema]

That's the way this sample is built. I wanted to show an example of developing an application protocol, to show how the various API can be used. The application protocol in the sample is completely minimal: the client asks for a file, the server sends it. If you want something more complex, you have to clone the sample code and change it to implement your own protocol design.

[jeromelau1984]

@huitema Thanks to answer! Another way, can you provide any sample code for client and server trans data of bidir transmission? Cause that I don't understand how to use cnx with bidir transmission.

[huitema]

It really is an application protocol issue. You could define your application as "open chat line", opening a bidir stream, and then have both side write to their side of the stream and read the other side. If both side mark their stream as "active", they will both receive callbacks when it is time to send, or when data is received from the peer. It would behave like a TCP connection.

[jeromelau1984]

OK, thanks. Let me try to implement that.

[jeromelau1984]

@huitema Thank you for your patience, it's working. One more question, can "openssl + picotls" be replaced with "mbedtls"? Whether the draft or RFC allows this.

[huitema]

@jeromelau1984 let's discuss that in a dedicated issue #1324.