Closed raphaelcoeffic closed 3 years ago
Some numbers on compression. I used pictures made by the cam and downloaded the ZIP file compressed (DEFLATED) and uncompressed.
% curl http://localhost:5000/data/download/photos.zip > photos.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1817k 0 1817k 0 0 28.1M 0 --:--:-- --:--:-- --:--:-- 28.1M
% zipinfo photos.zip
Archive: photos.zip
Zip file size: 1861104 bytes, number of entries: 11
-rw-r--r-- 2.0 unx 148920 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-30.jpg
-rw-r--r-- 2.0 unx 181267 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-20.jpg
-rw-r--r-- 2.0 unx 183212 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-10.jpg
-rw-r--r-- 2.0 unx 182620 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-00.jpg
-rw-r--r-- 2.0 unx 185654 bl defN 21-Mar-18 19:10 photos/2021-03-18-18-10-48.jpg
-rw-r--r-- 2.0 unx 182663 bl defN 21-Mar-18 19:10 photos/2021-03-18-18-10-38.jpg
-rw-r--r-- 2.0 unx 96640 bl defN 21-Mar-15 23:21 photos/2021-03-15-22-21-40.jpg
-rw-r--r-- 2.0 unx 152776 bl defN 21-Mar-15 20:40 photos/2021-03-15-19-40-55.jpg
-rw-r--r-- 2.0 unx 174100 bl defN 21-Mar-15 20:25 photos/2021-03-15-19-25-25.jpg
-rw-r--r-- 2.0 unx 193140 bl defN 21-Mar-15 20:24 photos/2021-03-15-19-24-55.jpg
-rw-r--r-- 2.0 unx 178833 bl defN 21-Mar-15 20:24 photos/2021-03-15-19-24-25.jpg
11 files, 1859825 bytes uncompressed, 1857738 bytes compressed: 0.1%
% curl http://localhost:5000/data/download/photos.zip > photos.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1819k 0 1819k 0 0 253M 0 --:--:-- --:--:-- --:--:-- 253M
% zipinfo photos.zip
Archive: photos.zip
Zip file size: 1861519 bytes, number of entries: 11
-rw-r--r-- 2.0 unx 148920 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-30.jpg
-rw-r--r-- 2.0 unx 181267 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-20.jpg
-rw-r--r-- 2.0 unx 183212 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-10.jpg
-rw-r--r-- 2.0 unx 182620 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-00.jpg
-rw-r--r-- 2.0 unx 185654 bl stor 21-Mar-18 19:10 photos/2021-03-18-18-10-48.jpg
-rw-r--r-- 2.0 unx 182663 bl stor 21-Mar-18 19:10 photos/2021-03-18-18-10-38.jpg
-rw-r--r-- 2.0 unx 96640 bl stor 21-Mar-15 23:21 photos/2021-03-15-22-21-40.jpg
-rw-r--r-- 2.0 unx 152776 bl stor 21-Mar-15 20:40 photos/2021-03-15-19-40-55.jpg
-rw-r--r-- 2.0 unx 174100 bl stor 21-Mar-15 20:25 photos/2021-03-15-19-25-25.jpg
-rw-r--r-- 2.0 unx 193140 bl stor 21-Mar-15 20:24 photos/2021-03-15-19-24-55.jpg
-rw-r--r-- 2.0 unx 178833 bl stor 21-Mar-15 20:24 photos/2021-03-15-19-24-25.jpg
11 files, 1859825 bytes uncompressed, 1859825 bytes compressed: 0.0%
As one can see from the compression gain (0.1%
) and the speed impact (28.1M
vs 253M
), there is no point in using compression here. However, it is still very convenient to download all the files in 1 single ZIP file.
Now you can also download all videos in one shot:
~/Src/NaturewatchCameraServer(download-photos|✚2…) % wget http://192.168.1.225:5000/data/download/videos.zip
--2021-03-21 13:58:21-- http://192.168.1.225:5000/data/download/videos.zip
Connecting to 192.168.1.225:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘videos.zip’
videos.zip [ <=> ] 265,95M 263MB/s in 1,0s
2021-03-21 13:58:22 (263 MB/s) - ‘videos.zip’ saved [278868487]
~/Src/NaturewatchCameraServer(download-photos|✚2…) % zipinfo videos.zip
Archive: videos.zip
Zip file size: 278868487 bytes, number of entries: 16
-rw-r--r-- 2.0 unx 18399739 bl stor 21-Mar-21 12:03 2021-03-21-10-47-11.mp4
-rw-r--r-- 2.0 unx 16764772 bl stor 21-Mar-21 12:03 2021-03-21-10-46-55.mp4
-rw-r--r-- 2.0 unx 18503965 bl stor 21-Mar-21 12:03 2021-03-21-10-46-42.mp4
-rw-r--r-- 2.0 unx 18541216 bl stor 21-Mar-21 12:03 2021-03-21-10-46-27.mp4
-rw-r--r-- 2.0 unx 16043429 bl stor 21-Mar-21 12:02 2021-03-21-10-46-13.mp4
-rw-r--r-- 2.0 unx 16313169 bl stor 21-Mar-21 12:02 2021-03-21-10-46-01.mp4
-rw-r--r-- 2.0 unx 17811980 bl stor 21-Mar-21 12:02 2021-03-21-10-45-34.mp4
-rw-r--r-- 2.0 unx 16361798 bl stor 21-Mar-21 12:02 2021-03-21-10-45-21.mp4
-rw-r--r-- 2.0 unx 16551296 bl stor 21-Mar-21 12:02 2021-03-21-10-45-09.mp4
-rw-r--r-- 2.0 unx 18458142 bl stor 21-Mar-21 12:02 2021-03-21-10-44-46.mp4
-rw-r--r-- 2.0 unx 18554846 bl stor 21-Mar-21 12:01 2021-03-21-10-44-34.mp4
-rw-r--r-- 2.0 unx 17342140 bl stor 21-Mar-21 12:01 2021-03-21-10-44-21.mp4
-rw-r--r-- 2.0 unx 15801167 bl stor 21-Mar-21 12:01 2021-03-21-10-44-08.mp4
-rw-r--r-- 2.0 unx 18065055 bl stor 21-Mar-21 12:01 2021-03-21-10-43-48.mp4
-rw-r--r-- 2.0 unx 17345681 bl stor 21-Mar-21 12:01 2021-03-21-10-43-25.mp4
-rw-r--r-- 2.0 unx 18007862 bl stor 21-Mar-21 12:01 2021-03-21-10-43-11.mp4
16 files, 278866257 bytes uncompressed, 278866257 bytes compressed: 0.0%
~/Src/NaturewatchCameraServer(download-photos|✚2…) % ls -lh videos.zip
-rw-r--r-- 1 rco staff 266M Mar 21 13:58 videos.zip
Please note: the test runs locally on a macbook, not on a raspberry pi.
@mikevanis I think it's your turn now :-)
Here are the routes:
/data/download/videos.zip
: download all videos./data/download/photos.zip
: same for photos.Thanks @raphaelcoeffic, looks great. I'll have a look at this as soon as I get a chance. Hopefully in the next couple of days.
This MR adds a new route (
/data/download/photos.zip
) to download all photos.ATM, the route does not take any arguments and just ZIPs all the files in the
/data/photos
. It is however intended to have the client side pass an array of file names (already implemented this way inFileSaver.download_all_photos(paths)
).Please note that the ZIP file itself is not compressed at all, as it is useless with JPEG files (no gain at all with DEFLATED). Moreover, this uses only streams, which means that the ZIP file never touches the disk, and everything is done with only little buffers (16KB for now).
There is however one drawback: the technique used is available only from Python 3.5 on (see https://docs.python.org/3/library/zipfile.html and "unseekable streams").
Implements #50