andrewjjenkins
commented
2 years ago Heap snapshot after uploading 47 photos shows that the leak is plugged. The photos were way more than 4MB combined; this leftover is probably internal cache / not-yet-GCable for the bytes buffer.
Problem: A memory leak occurs when uploading photos to Nixplay. For high-ish res photos (20Megapixel-ish), this is enough to grow past 512MB for the entire program after a few hundred photos (causing a pod with default parameters to be killed in GKE).
Solution: Pprof snapshots demonstrated the problem was in uploadS3 and related to the multipart MIME encoder.
There are two problems, both are fixed.
The input body is not closed (io.Copy doesn't close it). This is the body of the response from Google Photos. We don't ever save the image to a file, we pipe directly from the HTTP GET of the googlephoto image, to the MIME encoder to put it into the body of the POST we do to upload it to S3. Not closing the input body could cause us to retain the connection to googlephotos (and prevent its reuse) and retain the bytes of the body (maybe).
The response body from nixplay/S3 is not closed. This is the response for the request that was the POST containing the image. If the response isn't closed, the request may be retained, including the request body, which is the entire contents of the image. I think this is the one more clearly indicated as the culprit from the heap dump.
After this, the heap dump does not show growth without bound, even when uploading 50 pictures.