interactionresearchstudio / NaturewatchCameraServer

A Python / OpenCV camera server to stream Pi camera content to a remote client through a website.
https://mynaturewatch.net
GNU General Public License v3.0
103 stars 48 forks source link

ZIP file streamer to download all photos #78

Closed raphaelcoeffic closed 3 years ago

raphaelcoeffic commented 3 years ago

This MR adds a new route (/data/download/photos.zip) to download all photos.

ATM, the route does not take any arguments and just ZIPs all the files in the /data/photos. It is however intended to have the client side pass an array of file names (already implemented this way in FileSaver.download_all_photos(paths)).

Please note that the ZIP file itself is not compressed at all, as it is useless with JPEG files (no gain at all with DEFLATED). Moreover, this uses only streams, which means that the ZIP file never touches the disk, and everything is done with only little buffers (16KB for now).

There is however one drawback: the technique used is available only from Python 3.5 on (see https://docs.python.org/3/library/zipfile.html and "unseekable streams").

Implements #50

raphaelcoeffic commented 3 years ago

Some numbers on compression. I used pictures made by the cam and downloaded the ZIP file compressed (DEFLATED) and uncompressed.

Compressed

% curl http://localhost:5000/data/download/photos.zip > photos.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1817k    0 1817k    0     0  28.1M      0 --:--:-- --:--:-- --:--:-- 28.1M

% zipinfo photos.zip 
Archive:  photos.zip
Zip file size: 1861104 bytes, number of entries: 11
-rw-r--r--  2.0 unx   148920 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-30.jpg
-rw-r--r--  2.0 unx   181267 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-20.jpg
-rw-r--r--  2.0 unx   183212 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-10.jpg
-rw-r--r--  2.0 unx   182620 bl defN 21-Mar-20 10:29 photos/2021-03-20-09-29-00.jpg
-rw-r--r--  2.0 unx   185654 bl defN 21-Mar-18 19:10 photos/2021-03-18-18-10-48.jpg
-rw-r--r--  2.0 unx   182663 bl defN 21-Mar-18 19:10 photos/2021-03-18-18-10-38.jpg
-rw-r--r--  2.0 unx    96640 bl defN 21-Mar-15 23:21 photos/2021-03-15-22-21-40.jpg
-rw-r--r--  2.0 unx   152776 bl defN 21-Mar-15 20:40 photos/2021-03-15-19-40-55.jpg
-rw-r--r--  2.0 unx   174100 bl defN 21-Mar-15 20:25 photos/2021-03-15-19-25-25.jpg
-rw-r--r--  2.0 unx   193140 bl defN 21-Mar-15 20:24 photos/2021-03-15-19-24-55.jpg
-rw-r--r--  2.0 unx   178833 bl defN 21-Mar-15 20:24 photos/2021-03-15-19-24-25.jpg
11 files, 1859825 bytes uncompressed, 1857738 bytes compressed:  0.1%

Uncompressed

% curl http://localhost:5000/data/download/photos.zip > photos.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1819k    0 1819k    0     0   253M      0 --:--:-- --:--:-- --:--:--  253M

% zipinfo photos.zip 
Archive:  photos.zip
Zip file size: 1861519 bytes, number of entries: 11
-rw-r--r--  2.0 unx   148920 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-30.jpg
-rw-r--r--  2.0 unx   181267 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-20.jpg
-rw-r--r--  2.0 unx   183212 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-10.jpg
-rw-r--r--  2.0 unx   182620 bl stor 21-Mar-20 10:29 photos/2021-03-20-09-29-00.jpg
-rw-r--r--  2.0 unx   185654 bl stor 21-Mar-18 19:10 photos/2021-03-18-18-10-48.jpg
-rw-r--r--  2.0 unx   182663 bl stor 21-Mar-18 19:10 photos/2021-03-18-18-10-38.jpg
-rw-r--r--  2.0 unx    96640 bl stor 21-Mar-15 23:21 photos/2021-03-15-22-21-40.jpg
-rw-r--r--  2.0 unx   152776 bl stor 21-Mar-15 20:40 photos/2021-03-15-19-40-55.jpg
-rw-r--r--  2.0 unx   174100 bl stor 21-Mar-15 20:25 photos/2021-03-15-19-25-25.jpg
-rw-r--r--  2.0 unx   193140 bl stor 21-Mar-15 20:24 photos/2021-03-15-19-24-55.jpg
-rw-r--r--  2.0 unx   178833 bl stor 21-Mar-15 20:24 photos/2021-03-15-19-24-25.jpg
11 files, 1859825 bytes uncompressed, 1859825 bytes compressed:  0.0%

As one can see from the compression gain (0.1%) and the speed impact (28.1M vs 253M), there is no point in using compression here. However, it is still very convenient to download all the files in 1 single ZIP file.

raphaelcoeffic commented 3 years ago

Now you can also download all videos in one shot:

~/Src/NaturewatchCameraServer(download-photos|✚2…) % wget http://192.168.1.225:5000/data/download/videos.zip
--2021-03-21 13:58:21--  http://192.168.1.225:5000/data/download/videos.zip
Connecting to 192.168.1.225:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘videos.zip’

videos.zip                                  [      <=>                                     ] 265,95M   263MB/s    in 1,0s    

2021-03-21 13:58:22 (263 MB/s) - ‘videos.zip’ saved [278868487]

~/Src/NaturewatchCameraServer(download-photos|✚2…) % zipinfo videos.zip
Archive:  videos.zip
Zip file size: 278868487 bytes, number of entries: 16
-rw-r--r--  2.0 unx 18399739 bl stor 21-Mar-21 12:03 2021-03-21-10-47-11.mp4
-rw-r--r--  2.0 unx 16764772 bl stor 21-Mar-21 12:03 2021-03-21-10-46-55.mp4
-rw-r--r--  2.0 unx 18503965 bl stor 21-Mar-21 12:03 2021-03-21-10-46-42.mp4
-rw-r--r--  2.0 unx 18541216 bl stor 21-Mar-21 12:03 2021-03-21-10-46-27.mp4
-rw-r--r--  2.0 unx 16043429 bl stor 21-Mar-21 12:02 2021-03-21-10-46-13.mp4
-rw-r--r--  2.0 unx 16313169 bl stor 21-Mar-21 12:02 2021-03-21-10-46-01.mp4
-rw-r--r--  2.0 unx 17811980 bl stor 21-Mar-21 12:02 2021-03-21-10-45-34.mp4
-rw-r--r--  2.0 unx 16361798 bl stor 21-Mar-21 12:02 2021-03-21-10-45-21.mp4
-rw-r--r--  2.0 unx 16551296 bl stor 21-Mar-21 12:02 2021-03-21-10-45-09.mp4
-rw-r--r--  2.0 unx 18458142 bl stor 21-Mar-21 12:02 2021-03-21-10-44-46.mp4
-rw-r--r--  2.0 unx 18554846 bl stor 21-Mar-21 12:01 2021-03-21-10-44-34.mp4
-rw-r--r--  2.0 unx 17342140 bl stor 21-Mar-21 12:01 2021-03-21-10-44-21.mp4
-rw-r--r--  2.0 unx 15801167 bl stor 21-Mar-21 12:01 2021-03-21-10-44-08.mp4
-rw-r--r--  2.0 unx 18065055 bl stor 21-Mar-21 12:01 2021-03-21-10-43-48.mp4
-rw-r--r--  2.0 unx 17345681 bl stor 21-Mar-21 12:01 2021-03-21-10-43-25.mp4
-rw-r--r--  2.0 unx 18007862 bl stor 21-Mar-21 12:01 2021-03-21-10-43-11.mp4
16 files, 278866257 bytes uncompressed, 278866257 bytes compressed:  0.0%
~/Src/NaturewatchCameraServer(download-photos|✚2…) % ls -lh videos.zip 
-rw-r--r--  1 rco  staff   266M Mar 21 13:58 videos.zip

Please note: the test runs locally on a macbook, not on a raspberry pi.

raphaelcoeffic commented 3 years ago

@mikevanis I think it's your turn now :-)

Here are the routes:

mikevanis commented 3 years ago

Thanks @raphaelcoeffic, looks great. I'll have a look at this as soon as I get a chance. Hopefully in the next couple of days.