HTTP AT Commands

[AntonDrubetskiy]

Hi guys! Tell me how I can use HTTP AT commands to upload a file to the server (image, audio, or just a binary file)?

[ustccw]

@AntonDrubetskiy please take a look: https://docs.espressif.com/projects/esp-at/en/latest/AT_Command_Set/HTTP_AT_Commands.html#at-httpcpost-post-http-data-of-specified-length

[AntonDrubetskiy]

Thanks for the answer! I have looked at this command and tried to use it. However, this command uses Content-Type: text/xml, however I need to use Content-Type: multipart/form-data. Question: can I change the Content-Type in this command?

[ustccw]

@AntonDrubetskiy set the optional parameter <http_req_header> for the command.

[AntonDrubetskiy]

@ustccw Thank you!) Please show me an example , otherwise it is not clear how to do it in the description of AT commands

[AntonDrubetskiy]

@ustccw Hi! I figured out how to form a custom request header. However, it doesn't help me very much for uploading files to the server))

The developers of the server to which the data needs to be sent have provided the required header type:

POST /http_path HTTP/1.1 Host: example.com Content-Type: multipart/form-data; Content-Disposition: form-data; name = "image"; filename = "example.jpg" Content-Type: image/jpeg

The string POST/http_path HTTP/1.1 is generated by ESP32 based on the URL entered in the main part of the command. The rest of the lines are . Thus, a fully formed AT command for sending data to the server looks like this:

AT+HTTPCPOST="http://example.com/http_path",57118,4,"Host: example.com","Content-Type: multipart/form-data","Content-Disposition: form-data; name=image; filename=1.jpg","Content-Type: image/jpeg"

As you can see from the command, the file size is 57118 bytes.

After receiving a positive response:

OK

>

I am transferring files. After transferring the file, a message about successful transfer is received:

SEND OK

In this case, the server replies that the files could not be loaded.

According to the server administrators, the POST request occurs, but the files are not loaded.

For a more complete understanding of what is transmitted to the server, I created on the local computer using the HFS program http://www.rejetto.com/hfs/ and captured the data using Wireshark. The following URL is used http://192.168.x.xxx/test_folder/ When trying to download files using the above AT command, files also do not appear, but the server writes that a POST request has occurred. At the same time, Wireshark shows the following structure of the HTTP packet:

Hypertext Transfer Protocol POST /test_folder/HTTP/1.1\r\n User-Agent: ESP32 HTTP Client/1.0\r\n Host: 192.168.x.xxx\r\n Content-Type: image/jpeg\r\n Content-Disposition: form-data; name = image; filename = 1.jpg\r\n Content-Length: 57118\r\n \r\n [Full request URI: http://192.168.x.xxx/test_folder/] [HTTP request 1/2] [Response in frame: 1228] File Data: 57118 bytes

From this I can judge that the HTTP header is formed in accordance with the requirements of the server developers, but the files are not loaded.

Tell me, maybe I'm making some mistake that I don't notice? Does anyone have really working examples of uploading files to the HTTP server using the ESP32 AT commands?

[ustccw]

@AntonDrubetskiy Could you please provide the wireshark packets, detailed logs, and possible image you'd send.

Further, could i test your case, if yes, plz give me the server ip, port, and the image.

[AntonDrubetskiy]

@ustccw Hi!

http://194.0.91.158/HFS_Test_folder/

Port 80

[AntonDrubetskiy]

@ustccw

Transmitted packet:

No. Time    Source  Destination Protocol    Length  Info
1040    106.490791  192.168.1.120   192.168.1.124   HTTP    1012    POST /HFS_Test_folder/ HTTP/1.1  (JPEG JFIF image)

Its summary:

Frame 1040: 1012 bytes on wire (8096 bits), 1012 bytes captured (8096 bits) on interface \Device\NPF_{7B191864-8D96-451C-BB43-2CF9C3C05E05}, id 0
Ethernet II, Src: Espressi_af:99:88 (7c:df:a1:af:99:88), Dst: LiteonTe_31:ab:6d (94:08:53:31:ab:6d)
Internet Protocol Version 4, Src: 192.168.1.120, Dst: 192.168.1.124
Transmission Control Protocol, Src Port: 56955, Dst Port: 182, Seq: 56365, Ack: 1, Len: 958
[20 Reassembled TCP Segments (57322 bytes): #1005(204), #1006(2880), #1008(5760), #1010(5760), #1012(5760), #1014(1440), #1016(1440), #1017(2880), #1019(1440), #1021(4320), #1023(1440), #1025(4320), #1027(1440), #1029(4320), #1031(1440), #1]
Hypertext Transfer Protocol
JPEG File Interchange Format

Its expanded content:

Frame 1040: 1012 bytes on wire (8096 bits), 1012 bytes captured (8096 bits) on interface \Device\NPF_{7B191864-8D96-451C-BB43-2CF9C3C05E05}, id 0
    Interface id: 0 (\Device\NPF_{7B191864-8D96-451C-BB43-2CF9C3C05E05})
        Interface name: \Device\NPF_{7B191864-8D96-451C-BB43-2CF9C3C05E05}
        Interface description: Wi-Fi
    Encapsulation type: Ethernet (1)
    Arrival Time: Nov  2, 2021 12:15:32.018956000 Финляндия (зима)
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1635848132.018956000 seconds
    [Time delta from previous captured frame: 0.008118000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 106.490791000 seconds]
    Frame Number: 1040
    Frame Length: 1012 bytes (8096 bits)
    Capture Length: 1012 bytes (8096 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:tcp:http:image-jfif]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]

Ethernet II, Src: Espressi_af:99:88 (7c:df:a1:af:99:88), Dst: LiteonTe_31:ab:6d (94:08:53:31:ab:6d)
    Destination: LiteonTe_31:ab:6d (94:08:53:31:ab:6d)
        Address: LiteonTe_31:ab:6d (94:08:53:31:ab:6d)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: Espressi_af:99:88 (7c:df:a1:af:99:88)
        Address: Espressi_af:99:88 (7c:df:a1:af:99:88)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)

Internet Protocol Version 4, Src: 192.168.1.120, Dst: 192.168.1.124
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 998
    Identification: 0x0069 (105)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment Offset: 0
    Time to Live: 255
    Protocol: TCP (6)
    Header Checksum: 0x3364 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 192.168.1.120
    Destination Address: 192.168.1.124

Transmission Control Protocol, Src Port: 56955, Dst Port: 182, Seq: 56365, Ack: 1, Len: 958
    Source Port: 56955
    Destination Port: 182
    [Stream index: 34]
    [TCP Segment Len: 958]
    Sequence Number: 56365    (relative sequence number)
    Sequence Number (raw): 1273192767
    [Next Sequence Number: 57323    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 1022234194
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
        [TCP Flags: ·······AP···]
    Window: 5760
    [Calculated window size: 5760]
    [Window size scaling factor: -2 (no window scaling used)]
    Checksum: 0xae10 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [SEQ/ACK analysis]
        [iRTT: 0.003922000 seconds]
        [Bytes in flight: 958]
        [Bytes sent since last PSH flag: 57118]
    [Timestamps]
        [Time since first frame in this TCP stream: 0.087836000 seconds]
        [Time since previous frame in this TCP stream: 0.008118000 seconds]
    TCP payload (958 bytes)
    TCP segment data (958 bytes)

[20 Reassembled TCP Segments (57322 bytes): #1005(204), #1006(2880), #1008(5760), #1010(5760), #1012(5760), #1014(1440), #1016(1440), #1017(2880), #1019(1440), #1021(4320), #1023(1440), #1025(4320), #1027(1440), #1029(4320), #1031(1440), #1]
    [Frame: 1005, payload: 0-203 (204 bytes)]
    [Frame: 1006, payload: 204-3083 (2880 bytes)]
    [Frame: 1008, payload: 3084-8843 (5760 bytes)]
    [Frame: 1010, payload: 8844-14603 (5760 bytes)]
    [Frame: 1012, payload: 14604-20363 (5760 bytes)]
    [Frame: 1014, payload: 20364-21803 (1440 bytes)]
    [Frame: 1016, payload: 21804-23243 (1440 bytes)]
    [Frame: 1017, payload: 23244-26123 (2880 bytes)]
    [Frame: 1019, payload: 26124-27563 (1440 bytes)]
    [Frame: 1021, payload: 27564-31883 (4320 bytes)]
    [Frame: 1023, payload: 31884-33323 (1440 bytes)]
    [Frame: 1025, payload: 33324-37643 (4320 bytes)]
    [Frame: 1027, payload: 37644-39083 (1440 bytes)]
    [Frame: 1029, payload: 39084-43403 (4320 bytes)]
    [Frame: 1031, payload: 43404-44843 (1440 bytes)]
    [Frame: 1033, payload: 44844-49163 (4320 bytes)]
    [Frame: 1035, payload: 49164-50603 (1440 bytes)]
    [Frame: 1037, payload: 50604-52043 (1440 bytes)]
    [Frame: 1038, payload: 52044-56363 (4320 bytes)]
    [Frame: 1040, payload: 56364-57321 (958 bytes)]
    [Segment count: 20]
    [Reassembled TCP length: 57322]
    [Reassembled TCP Data: 504f5354202f4846535f546573745f666f6c6465722f20485454502f312e310d0a557365…]

Hypertext Transfer Protocol
    POST /HFS_Test_folder/ HTTP/1.1\r\n
    User-Agent: ESP32 HTTP Client/1.0\r\n
    Host: 192.168.1.124:182\r\n
    Content-Type: image/jpeg\r\n
    Content-Disposition: form-data; name=image; filename=1.jpg\r\n
    Content-Length: 57118\r\n
    \r\n
    [Full request URI: http://192.168.1.124:182/HFS_Test_folder/]
    [HTTP request 1/2]
    [Response in frame: 1048]
    File Data: 57118 bytes

JPEG File Interchange Format
    Marker: Start of Image (0xffd8)
    Marker segment: Reserved for application segments - 0 (0xFFE0)
        Marker: Reserved for application segments - 0 (0xffe0)
        Length: 16
        Identifier: JFIF
        Version: 1.1
            Major Version: 1
            Minor Version: 1
        Units: Dots per inch (1)
        Xdensity: 96
        Ydensity: 96
        Xthumbnail: 0
        Ythumbnail: 0
    Marker segment: Reserved for application segments - 1 (0xFFE1)
        Marker: Reserved for application segments - 1 (0xffe1)
        Length: 34
        Identifier: Exif
        Endianness: big endian
        Start offset of IFD starting from the TIFF header start: 8 bytes
        Number of fields in this IFD: 1
        Exif Tag: Orientation (274)
        Type: SHORT (3)
        Count: 1
        Value offset from start of TIFF header: 65536
        Offset to next IFD from start of TIFF header: 0 bytes
    Marker segment: Define quantization table(s) (0xFFDB)
        Marker: Define quantization table(s) (0xffdb)
        Length: 67
        Remaining segment data: 65 bytes
    Marker segment: Define quantization table(s) (0xFFDB)
        Marker: Define quantization table(s) (0xffdb)
        Length: 67
        Remaining segment data: 65 bytes
    Start of Frame header: Start of Frame (non-differential, Huffman coding) - Baseline DCT (0xFFC0)
        Marker: Start of Frame (non-differential, Huffman coding) - Baseline DCT (0xffc0)
        Length: 17
        Sample Precision (bits): 8
        Lines: 648
        Samples per line: 1152
        Number of image components in frame: 3
        Component identifier: 1
        0010 .... = Horizontal sampling factor: 2
        .... 0010 = Vertical sampling factor: 2
        Quantization table destination selector: 0
        Component identifier: 2
        0001 .... = Horizontal sampling factor: 1
        .... 0001 = Vertical sampling factor: 1
        Quantization table destination selector: 1
        Component identifier: 3
        0001 .... = Horizontal sampling factor: 1
        .... 0001 = Vertical sampling factor: 1
        Quantization table destination selector: 1
    Marker segment: Define Huffman table(s) (0xFFC4)
        Marker: Define Huffman table(s) (0xffc4)
        Length: 31
        Remaining segment data: 29 bytes
    Marker segment: Define Huffman table(s) (0xFFC4)
        Marker: Define Huffman table(s) (0xffc4)
        Length: 181
        Remaining segment data: 179 bytes
    Marker segment: Define Huffman table(s) (0xFFC4)
        Marker: Define Huffman table(s) (0xffc4)
        Length: 31
        Remaining segment data: 29 bytes
    Marker segment: Define Huffman table(s) (0xFFC4)
        Marker: Define Huffman table(s) (0xffc4)
        Length: 181
        Remaining segment data: 179 bytes
    Start of Segment header: Start of Scan (0xFFDA)
        Marker: Start of Scan (0xffda)
        Length: 12
        Number of image components in scan: 3
        Scan component selector: 1
        0000 .... = DC entropy coding table destination selector: 0
        .... 0000 = AC entropy coding table destination selector: 0
        Scan component selector: 2
        0001 .... = DC entropy coding table destination selector: 1
        .... 0001 = AC entropy coding table destination selector: 1
        Scan component selector: 3
        0001 .... = DC entropy coding table destination selector: 1
        .... 0001 = AC entropy coding table destination selector: 1
        Start of spectral or predictor selection: 0
        End of spectral selection: 63
        0000 .... = Successive approximation bit position high: 0
        .... 0000 = Successive approximation bit position low or point transform: 0
    Entropy-coded segment (dissection is not yet implemented): fdfca28a2800a28a2800a28a2800a28a2800a28a2800a28a2800a28a2800a28a2800a28a…
    Marker: End of Image (0xffd9)

[ustccw]

i tried to post the image by postman, but failed to connect to the server, maybe firewall or some else.

Could you please test your POST request with the http tool: postman. If it works, then should capture the http packet, and compare to the http packet which Wi-Fi module sent, find out what is the difference; If not, then should find out what is wrong with the post request.

[AntonDrubetskiy]

Hi!

We made a POST request through Postman

It works with this header:

POST / HTTP/1.1
User-Agent: PostmanRuntime/7.28.4
Accept: /
Postman-Token: 4625410d-22f9-4764-ab29-634d00b209fa
Host: 127.0.0.1
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------------------------625776993078715828846231
Cookie: HFS_SID_=0.834543708479032
Content-Length: 93955

By the way, I have flashed my ESP32 device with the latest firmware version:

AT+GMR
AT version:2.3.0.0-dev(3f6a1cd - ESP32C3 - Oct 19 2021 09:32:17)
SDK version:v4.3.1
compile time(d7c8721):Nov  3 2021 12:21:44
Bin version:2.2.0(MINI-1)

[OleksandrDroid]

Some additional explanations.

We have two kinds of headers that were obtained and tested by Postman. This one

POST / HTTP/1.1 User-Agent: PostmanRuntime/7.28.4 Accept: / Postman-Token: 4625410d-22f9-4764-ab29-634d00b209fa Host: 127.0.0.1 Accept-Encoding: gzip, deflate, br Connection: keep-alive Content-Type: multipart/form-data; boundary=--------------------------625776993078715828846231 Cookie: HFSSID=0.834543708479032 Content-Length: 93955

and this one:

POST / HTTP/1.1 User-Agent: PostmanRuntime/7.28.4 Accept: / Postman-Token: c46f1698-887b-4def-8050-73f01a9b661d Host: 127.0.0.1 Accept-Encoding: gzip, deflate, br Connection: keep-alive Cookie: HFSSID=0.834543708479032 Content-Length: 93748 Content-Type: image/png

The first header we obtain if in Postman on the "body" tab I choose "form-data" option and there I put a needed file. It works perfect but we can not reproduce the same with ESP because after adding "Content-Type: multipart/form-data; boundary=--------------------------625776993078715828846231" it returns ERROR.

The second header we obtain if in Postman on the "body" tab I choose "binary" option and there I put a needed file. This request executes OK but really the file does not appear on the server. Also, we can use this header successfully with ESP (it does not return ERROR) but the result is the same - the transaction is OK but the file does not appear on the server.

Our questions are following: 1) do you have source code that demonstrates how to upload a file to the HTTP server using a POST request? 2) how to send correctly with ESP a "Content-Type: multipart/form-data; boundary=--------------------------625776993078715828846231" header?

[ustccw]

@AntonDrubetskiy AT HTTP Post request commands are like the following:

AT+RST

OK

ready
AT+CWMODE=1

OK
AT+CWJAP="688018",""
WIFI CONNECTED
WIFI GOT IP

OK
AT+HTTPCPOST="http://194.0.91.158/HFS_Test_folder/",57118,2,"Content-Type: multipart/form-data; boundary=--------------------------625776993078715828846231","Cookie: HFS_SID_=0.834543708479032"

OK

>
SEND FAIL
AT+GMR
AT version:2.3.0.0-dev(3f6a1cd - ESP32C3 - Oct 19 2021 09:32:17)
SDK version:v4.3.1-245-gba15ac8634
compile time(d7c8721):Nov  1 2021 11:51:13
Bin version:2.2.0(MINI-1)

OK

see the doc for more details.

i guess the the total length of AT command you input is bigger than 256 bytes, it returns ERROR.
If yes, consider to reduce command length.

[OleksandrDroid]

Hi guys.

I really apologize, but we don't need a guess, we need a solution. We are developing a device with an annual volume of 20K units and cannot get support for several weeks on the basic functions of your device. It `s so sad.

We have provided information that you needed about connection to the server and by Postman application we have found the correct header.

Please tell me what else we need to provide to get an exhaustive answer to our question?

We are ready to cooperate because we are stuck in this project.

[AntonDrubetskiy]

@ustccw Hi!

The problem was solved in another way.

We used a TCP connection to send the header and data. This is how it looks for sending, for example, a text file named Hello.txt:

POST / HFS_Test_folder / HTTP / 1.1
Host: 192.168.1.124
Connection: keep-alive
Content-Length: 10
Cache-Control: max-age = 0
Content-Type: multipart / form-data; boundary = ---- WebKitFormBoundary9SAa7m1sBwTIhcuV
User-Agent: ESP32 HTTP Client / 1.0
Accept: text / html, application / xhtml + xml, application / xml; q = 0.9, image / avif, image / webp, image / apng, * / *; q = 0.8, application / signed-exchange; v = b3; q = 0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: ru, en-US; q = 0.9, en; q = 0.8
Cookie: HFS_SID_ = 0.219766394933686

------ WebKitFormBoundary9SAa7m1sBwTIhcuV
Content-Disposition: form-data; name = "file"; filename = "Hello.txt"
Content-Type: text / plain

1234567890
------ WebKitFormBoundary9SAa7m1sBwTIhcuV
Content-Disposition: form-data; name = "file"; filename = ""
Content-Type: application / octet-stream

------ WebKitFormBoundary9SAa7m1sBwTIhcuV--

Where 1234567890 is the data that is in the file.

I took an example of a header from capturing an HTTP packet using Wireshark when uploading a file from localhost to an HFS server. Probably, for other tasks the header will differ from the one above.

We used the following commands to send the file:

AT + CWJAP = "My_WIFI", "My_WIFI_Password"

WIFI CONNECTED
WIFI GOT IP

OK

AT + CIPSTART = "TCP", "192.168.1.124", 80

CONNECT

OK

AT + CIPMODE = 0

OK AT + CIPSEND = 829

OK

>
Recv 829 bytes

SEND OK

I also want to say that to simplify sending the header, we wrote it in a text file and after the command AT + CIPSEND = 829 sent it through the terminal. For this, CoolTerm was used. In this example, 829 bytes is the size of the text file that we sent via CoolTerm. The sent file contains a header with the data presented above.

[ustccw]

@OleksandrDroid I'm very sorry for the inconvenience. Our example documentation is really not rich enough for you to use quickly. In addition, GitHub issues is used for bug report and feature request. If you need to have a real-time support, please contact our business here~

[ustccw]

@AntonDrubetskiy yes, it is a easy way to establish the TCP connection and construct the HTTP packets. In additional, you can use AT+CIPSENDL command to send the big files in this way.

[AntonDrubetskiy]

@ustccw Thanks for the information)))