File uploads via webui are corrupted

[rreiner]

Info

Which version of Pi are you using: Zero W Which github revision of software: April 2023 Release - v23.04.01 Which board version: FULL Which computer is the PiSCSI connected to: NA

Describe the issue

All file uploads via the webui are corrupted. The webui's file size check correctly detects this and flags the upload with "Transferred file corrupted!".

The files uploaded via the webui onto the Pi are in each case slightly larger than the original.

Uploads were performed from Chrome 112.0.5615.138 (Official Build) (64-bit) (cohort: Stable) on Windows 10.

Examples (the files in /home/pi/images being the webui uploads and those in /var/tmp having been copied with scp for comparison):

/home/pi/images/nextstep-3-3-patches-NSOSY2K.iso /var/tmp/nextstep-3-3-patches-NSOSY2K.iso differ: byte 89128961, line 509690
-rw-r--r-- 1 root    root    93184008 Apr 30 12:36 /home/pi/images/nextstep-3-3-patches-NSOSY2K.iso
-rwxr-x--- 1 rreiner rreiner 90142720 Apr 30 12:35 /var/tmp/nextstep-3-3-patches-NSOSY2K.iso

/home/pi/images/sol_2.5.1_sparc.iso /var/tmp/sol_2.5.1_sparc.iso differ: byte 4194305, line 16156
-rw-r--r-- 1 root    root    367722596 Apr 30 12:17 /home/pi/images/sol_2.5.1_sparc.iso
-rwxr-x--- 1 rreiner rreiner 365608960 Apr 30 12:30 /var/tmp/sol_2.5.1_sparc.iso

[rdmark]

So, I can't reproduce your issue. I'm using the same April release code, and upload two large files (647MB, 500MB) via the web ui dropzone form. Both transferred files have the exact same byte size as the original file. This with the latest Vivaldi browser on Win10.

I can't immediately think of a good reason this is happening for you. Rare browser bug? Something that messes with the packet headers on your network?

[rreiner]

Hmm.

• It can't be the network, as 1) other error-checked communications between the same two hosts (Windows desktop and the PiSCSI Pi are fine, e.g. ssh / scp / sftp, and 2) there are no other network problems

• It can't be a Chrome bug, as it also happens with Firefox.

Some more new information:

• The corruption is not random -- precisely the same corrupted bits arrive each time if I upload multiples copies of the same ISO.

I'm trying to find some pattern in the corruption. So far I can see that the offset of the first corrupted byte is not always the same in different files (but it is the same in multiple uploaded copies of the same file).

[rdmark]

The file uploading is 99% the dropzone.js library, with a thin python code layer that handles the chunking and writing the file to disk. We're using the latest stable dropzone library, so my assumption is that something goes wrong in the python layer. The chunks get messed up somehow along the process. What version of the python interpreter are you running?

[rreiner]

The Pi is newly installed from the April binary distro of PiSCSI with all Raspbian updates applied. So we have

$ python -V
Python 3.9.2

[rreiner]

Using the rather nice vbindiff tool (https://github.com/madsen/vbindiff) I've found that on my first test ISO, the corruptions are as follows:

0x00400000 - insert CR LF
0x00500002 - insert CR LF
0x00600004 - insert CR LF
0x00700006 - insert CR LF
0x00800008 - insert CR LF
and so on, until
0x0680884C - insert a large chunk of garbage

On a second test file, they are as follows:

0x05500000 - insert CR LF
0x055f7802 - insert a large chunk of garbage

[antoniograzioli]

same here. most of the times not always, but 90% of the time I try to upload CD-Rom Images (500-600MB) using the Web Ui and I get a "Transferred file corrupted!" error. Smaller files (tested with +/-100MB) usually works...

[rdmark]

@antoniograzioli Is it the same for you as the original reporter, that the corruption is consistent and predictable, that uploading a particular file always leads to a file with the wrong size?

[antoniograzioli]

Actually no, the same file sometimes gets uploaded correctly, sometimes it gets corrupted. I need to check back tomorrow, tonight my network is not really stable and my Mac sometime disconnects form the router...

[rdmark]

Got it. Then it may be a different issue. Although the ultimate solution may be the same. Is the resulting file larger or smaller than it should be?

[rdmark]

Since I can't reproduce the issue myself (although I'll keep trying) one thing you both can play around with in the meantime is the parameters that we pass to dropzone:

https://github.com/PiSCSI/piscsi/blob/57fa874d53b0a8e5686e3c408eb6fdad6ddf3d7d/python/web/src/templates/upload.html#L42

The upload.html template has the code that controls f.e. the chunk retry and chunk size settings. I'd try turning off retries (to see if the transfer fails instead) or adjust the chunk size upwards or downwards.

[rdmark]

One more question: Are you all uploading one file at a time, or multiple in parallel?

[rreiner]

In my case it corrupts them both when uploading one and when uploading multiple at a time.

And also both when I drop files on the page and when I select then by clicking and them choosing in the file picker.

[rdmark]

@antoniograzioli What RPi hardware are you using? Is it connected to your LAN via ethernet or wifi?

[rdmark]

BTW I should add, in case either of you tries to hack upload.html (or the python code) -- you have to either restart piscsi-web after changing the code, or run piscsi-web in developer mode. sudo ./start.sh --dev-mode

[antoniograzioli]

Raspberry Pi 3 Model B Plus Rev 1.3, Linux 6.1.21-v8+ aarch64

using Ethernet cables for both my Mac and the RPI

Il giorno mar 2 mag 2023 alle 00:51 Daniel Markstedt < @.***> ha scritto:

@antoniograzioli https://github.com/antoniograzioli What RPi hardware are you using? Is it connected to your LAN via ethernet or wifi?

— Reply to this email directly, view it on GitHub https://github.com/PiSCSI/piscsi/issues/1157#issuecomment-1530487991, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2RCCZ6IHTZNOPPOEBFDDDXEA46FANCNFSM6AAAAAAXRA4JKY . You are receiving this because you were mentioned.Message ID: @.***>

[rreiner]

RPi Zero W, connected over wifi. The uploader machine is connected via gigabit ethernet.

Should add that the same occurs identically on both my PiSCSI units (they are the same hardware and the same software, so I suppose that's not surprising, but it does exclude bizarre satisfies-the-TCP-checksum-but-somehow-corrupts-the-file-anyway hardware faults.)

[rreiner]

Some upload experiments -- first with unmodified upload.html, with default chunksize == 1048576 (i.e. 1MB)

1/

• created a test file containing exactly 100000000 'a' characters
• when uploaded to the Pi this is corrupted and expands to 101156038 bytes
• Using `vbindiff`` I see the following corruption:

  0x00000000 insert CRLF
  0x00100002 insert CRLF -- that's 1MB plus the two inserted bytes
  0x00200004 insert CRLF
  and so on, until:
  0x5f5E156 insert 1MB of 'a' characters

2/

• created a test file containing 100M \0 characters
• when uploaded this expands identically to 101156038 characters with the same corruption pattern,

So a) the bug does not appear data dependent, and b) the corruption appears every chunksize bytes.

3/ Using upload.html modified so chunksize is 2MiB rather than the default 1MiB:

• the corruption is now different -- the file expands to 104301670 bytes rather than the previous 101156038
• Using vbindiff I see the following:

  0x00000000 insert CRLF
  0x00200002 insert CRLF
  0x00400004 insert CRLF
  and so on, until:
  0x05F5E100: insert 2MB of 'a' characters

So, confirmed: the corruption consists of

• a CR LF pair inserted before each chunk, plus
• one chunksize of data from the file appended to the end

4/ Trying upload.html with chunking: false and forceChunking: false:

• the upload never completes -- progress bar reaches full and hangs there

5/ Trying upload.html with parallelChunkUploads: true and chunksize back to 1MiB:

• The corruption is different. File now expands to 100107460 bytes
• I see the same pattern of corruption as in case 1 above but there is less junk appended at the end.

That's all I have time for now, hopefully this is helpful -- although it looks increasingly like it's a dropzone.js bug.

[rdmark]

@rreiner Thanks for the thorough testing! The unexpected CRLF is a good hint to what's going on. My understanding is that http chunked transfer encoding inserts CRLF after each chunk per spec, as the delineator, but I expect them to be stripped when reassembling the file. Perhaps we need to add some handling of this to the python code. Let me do some research.

[rdmark]

@rreiner Would you be able to test if this fixes it for you? https://github.com/PiSCSI/piscsi/pull/1267

(Sorry for the long wait. I was busy being a full-time job seeker, and then relocating to another continent!)

[rdmark]

As a parallel experiment, here is another PR that bumps the dz.js library to 6.0.0-beta1. https://github.com/PiSCSI/piscsi/pull/1271

It's a complete shot in the dark whether it will fix your issue or not, but reading their changelog I think we will want to upgrade anyways, since they deprecate Internet Explorer support, and drastically reduce the size of the library. (46kB vs 115kB)

[rreiner]

Yes, I'lll drag that machine up from the basement and test within the next couple of days.

On Wed, Oct 25, 2023, 06:52 Daniel Markstedt @.***> wrote:

@rreiner https://github.com/rreiner Would you be able to test if this fixes it for you? #1267 https://github.com/PiSCSI/piscsi/pull/1267

(Sorry for the long wait. I was busy being a full-time job seeker, and then relocating to another continent!)

— Reply to this email directly, view it on GitHub https://github.com/PiSCSI/piscsi/issues/1157#issuecomment-1779002497, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6TFYLD2KVWDLXPDRALQL3YBDVN3AVCNFSM6AAAAAAXRA4JK2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZZGAYDENBZG4 . You are receiving this because you were mentioned.Message ID: @.***>

[rreiner]

I've got access again to the relevant device. Given the discussion on #1267 please let me know what tests you would like me to carry out and with what version.

[rdmark]

@rreiner Wonderful! So for starters, to establish a baseline, please use the latest code in https://github.com/PiSCSI/piscsi/pull/1267 (i.e. check out the branch rdmark-issue-1157 )

Right now it only adds some logging and does nothing different to the uploaded chunks, so I expect the upload process to fail for you like before. However, it would be very helpful if you could capture a few logs so that we can see the actual chunk size and offsets that are coming through.

You may need to set the log level to info to see the logs.

[rreiner]

Well, the result is unexpected -- uploads no longer fail.

I did the following:

sudo systemctl stop piscsi
sudo systemctl stop piscsi-web
cd ~/piscsi
git fetch
git checkout rdmark-issue-1157
git pull
./easyinstall.sh   # selected option 1
# wait a couple of hours -- it's a Pi Zero
sudo systemctl start piscsi
sudo systemctl start piscsi-web

And uploads now work normally.

[rreiner]

I also tested on a fresh install using the November 2023 prerelease (git checkout v23.11.01). Uploads also works fine on that.

[rdmark]

Thanks for testing thoroughly! I refactored the upload code leading up to the release, which may have an impact for your corner case.

Please let us know if this issue rears its ugly head again. This is clearly a fragile part of the code. Since it went away unexpectedly it may come back unexpectedly...