Use heuristic when downloading/streaming file from jotta

[jkaberg]

As per chat,

@jkaberg, will you please create a new bug report about the jottafuse download speeds. and add the speedtest output for reference I've looked at other networked fuse file systems, and seen some use a heuristic to actually pull bigger chunks when it is determined that a process is reading the whole file. Because currently, 'jottafuse' is really dumb and pulls the file down piece by piece, when you really want the whole file

speedtest.py output

joel@dunder:/opt/jottalib/tests# python speedtest.py
Creating test file.
[################################] 1000/1000 - 00:00:00
Test 1 | requests | Upload | File size: 97.656MiB
Network upload speed 5.802MiB/sec  102400535/102400000 - 00:00:00

Test 2 | requests | Read | File size: 97.656MiB
Network download speed 7.116MiB/sec

Test 2 | requests | Stream | File size: 97.656MiB
Network download speed 6.974MiB/sec 02400000/102400000 - 09:06:52

Test 3 | urllib3 | Read | File size: 97.656MiB
Network download speed 7.038MiB/sec

Test 3 | urllib3 | Stream | File size: 97.656MiB
Network download speed 6.732MiB/sec0/102400000 - 00:00:00

Finished.
joel@dunder:/opt/jottalib/tests#

curl output when pipeing a file to /dev/null

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1281M  100 1281M    0     0  7497k      0  0:02:55  0:02:55 --:--:-- 7593k

rsync the same file with --stats output (mounted with jottafuse.py)

Number of files: 1
Number of files transferred: 1
Total file size: 1343768420 bytes
Total transferred file size: 1343768420 bytes
Literal data: 1343768420 bytes
Matched data: 0 bytes
File list size: 94
File list generation time: 0.028 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1343932596
Total bytes received: 31

sent 1343932596 bytes  received 31 bytes  1012378.63 bytes/sec
total size is 1343768420  speedup is 1.00

[havardgulldahl]

this requires a lot of work, so I'm not delaying 0.4 for it.

and frankly, I really want people to use the native jotta-* tools to communicate with JottaCloud, and not the fuse abstraction, because it's really hard to make that thing work as fast as everyone wants

[jkaberg]

Note to self

https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/internal/gcsx/random_reader.go#L236

[jkaberg]

Also this

https://github.com/yadayada/acd_cli/blob/master/acdcli/acd_fuse.py#L53

perhaps the above is what your looking for @havardgulldahl ?

And while I understand your reasoning, some scenarios requires a file system (however slow it might be)