s3_upload branch (bug?): Occasionally Unix new datetime fails to upload

[scottveirs]

This has only happened ~twice during experimental deployment of boto-based Python s3_upload branch, but player fails to play (no response on pressing play button at live.orcasound.net/ ) because Unix datetime stamped directory is created on Raspberry Pi4 but not transferred to S3 streaming bucket. Thus player finds the latest.txt file, but throws and error when trying to access the latest .m3u8 file.

Possible bug may be in the inotify logic and/or rare issues with boto failing?

[scottveirs]

Happened again tonight (Fri Jan 29, 2021 upon the 18:30 local container restart; Orcasound Lab Rpi4). Logged in via Dataplicity and confirmed with htop that stream.sh, ffmpeg, and s3_upload.py all running (but low CPU usage?). The datetime directory exists in /tmp but is empty.

Checked S3 and don't see new datetime "folder" object in streaming bucket. This suggests inotify didn't catch the directory creation event on Rpi. Why would that be?

I suspected the hw:pisound might be mapped to wrong device, but I think 1,0 is expected... If ffmpeg can't find the audio device, perhaps it never creates an m3u8 or ts segments in the tmp/.../datetime directory?

[scottveirs]

Noting another occurrence today (Jan 10, 2022 for bucket object 1641825019) on the orcasound-lab node with the same behavior as on 1/29/2021 above.

[scottveirs]

Val and I just observed another instance of this bug for the orcasound-lab node. The missing data directory in this case was 1703601018

[scottveirs]

And a (related, I think?) event for orcasound-lab node between the last two of these entries:

The missing datetime object means there was a a loss of 6 hours of data last night (5/4/24 18:30 - 5/5/24 00:30)

[mcshicks]

If you look in base/upload_s3.py there is a call that uploads the file. We don't actually get the return value from that call.

resource.meta.client.upload_file(uploadfile, BUCKET, uploadkey, ExtraArgs={'ACL': 'public-read'})

I just checked the API documentation and there is a return value True for success and False for failure to upload.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html

I'm not sure what the state of the logging, is there was some problem w/ the logging tool and raspbian. But assuming that the logging was working it would seem adding a check to this call would seem to me to be the next step.

Also even though the timestamp name looks like a folder, it's not really. There is no "create directory" function. So if we are missing a timestamp folder completely that's more like there was some error when stream.sh was called, but the container is supposed to restart if there is an error. So it's a bit of a puzzle.