search

venari / timelapse

A set of tools/scripts to automate the taking and creation of timelapse videos and videos with a Raspberry Pi
MIT License
1 stars 0 forks source link

Pending telemetry not getting uploaded #34

Closed leighghunt closed 2 years ago

leighghunt commented 2 years ago

Noticed that old telemetry ism't getting uploaded.

2022-07-28 22:22:56,430 ERROR: 2022-07-28 22:22:56.430234 uploadPendingTelemetry() failed.
2022-07-28 22:22:56,431 ERROR: Expecting value: line 1 column 1 (char 0)

logs/timelapse.log:

2022-07-28 22:22:56,426 INFO:  uploading 2022-07-10_040055.json
2022-07-28 22:22:56,427 DEBUG: telemetryTimestamp:
2022-07-28 22:22:56,427 DEBUG: 2022-07-10 04:00:55
2022-07-28 22:22:56,430 ERROR: 2022-07-28 22:22:56.430234 uploadPendingTelemetry() failed.
2022-07-28 22:22:56,431 ERROR: Expecting value: line 1 column 1 (char 0)
2022-07-28 22:22:56,432 INFO:  uploading 2022-07-28_222252.jpg
2022-07-28 22:22:56,438 DEBUG: imageTimestamp:
2022-07-28 22:22:56,438 DEBUG: 2022-07-28 22:22:52
2022-07-28 22:22:56,439 DEBUG: data:
2022-07-28 22:22:56,440 DEBUG: {'SerialNumber': '00000000638a42a2', 'Timestamp': '2022-07-28T22:22:52+12:00'}
2022-07-28 22:22:56,440 DEBUG: Posting image to API...
2022-07-28 22:22:56,458 DEBUG: Starting new HTTPS connection (1): timelapse-dev.azurewebsites.net:443
2022-07-28 22:22:57,333 DEBUG: https://timelapse-dev.azurewebsites.net:443 "POST /api/Image HTTP/1.1" 200 None
2022-07-28 22:22:57,338 DEBUG: Response code: 200
2022-07-28 22:22:57,338 DEBUG: Image uploaded successfully
2022-07-28 22:22:57,339 DEBUG: Response text:
2022-07-28 22:22:57,340 DEBUG: {
    "id": 111152,
    "timestamp": "2022-07-28T10:22:52Z",
    "blobUri": "https://venaritimelapse.blob.core.windows.net/timelapse/11_2022-07-28_222252.jpg",
    "deviceId": 11,
    "device": {
        "id": 11,
        "serialNumber": "00000000638a42a2",
        "name": "Pi Zero W 2 number 3",
        "description": "3rd camera"
    }
}
2022-07-28 22:23:07,447 INFO:  uploading 2022-07-10_040055.json
2022-07-28 22:23:07,449 DEBUG: telemetryTimestamp:
2022-07-28 22:23:07,450 DEBUG: 2022-07-10 04:00:55
2022-07-28 22:23:07,454 ERROR: 2022-07-28 22:23:07.454565 uploadPendingTelemetry() failed.
2022-07-28 22:23:07,455 ERROR: Expecting value: line 1 column 1 (char 0)
2022-07-28 22:23:17,546 INFO:  uploading 2022-07-10_040055.json
2022-07-28 22:23:17,548 DEBUG: telemetryTimestamp:
2022-07-28 22:23:17,548 DEBUG: 2022-07-10 04:00:55
2022-07-28 22:23:17,551 ERROR: 2022-07-28 22:23:17.551433 uploadPendingTelemetry() failed.
2022-07-28 22:23:17,552 ERROR: Expecting value: line 1 column 1 (char 0)
2022-07-28 22:23:23,180 DEBUG: beginning capture
2022-07-28 22:23:23,227 DEBUG: warming up camera...
ethandhunt commented 2 years ago

traceback.print_exc() at the end of the uploadPendingTelemetry returns

Traceback (most recent call last):                                                                                         
  File "uploadPending.py", line 109, in uploadPendingTelemetry                                                             
    api_data = json.load(open(pendingTelemetryFolder + telemetryFilename, 'rb'))                                                               
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load                                                            
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)                                              
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads                                                           
    return _default_decoder.decode(s)                                                                                      
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode                                                           
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())                                                                      
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode                                                       
    raise JSONDecodeError("Expecting value", s, err.value) from None                                                       
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ethandhunt commented 2 years ago

pendingTelemetryFolder + telemetryFilename evaluated to ../output/telemetry/pending/2022-07-10_040055.json

ethandhunt commented 2 years ago 
$ file 2022-07-10_040055.json                                                                                                          
2022-07-10_040055.json: empty
ethandhunt commented 2 years ago

it seems like the file is empty and that's making the script throw a json decoding error

ethandhunt commented 2 years ago

looking at the log surrounding the time that one of the empty files was created

2022-07-02 08:00:58,009 DEBUG: Logged to API.                                                                                                                                                    
2022-07-02 08:00:58,024 INFO:  uploading 2022-07-02_080053.jpg                                                                                                                                   
2022-07-02 08:00:58,033 DEBUG: imageTimestamp:                                                                                                                                                   
2022-07-02 08:00:58,034 DEBUG: 2022-07-02 08:00:53                                                                                                                                               
2022-07-02 08:00:58,035 DEBUG: data:                                                                                                                                                             
2022-07-02 08:00:58,036 DEBUG: {'SerialNumber': '00000000638a42a2', 'Timestamp': '2022-07-02T08:00:53+12:00'}                                                                                    
2022-07-02 08:00:58,037 DEBUG: Posting image to API...                                                                                                                                           
2022-07-02 08:00:58,052 DEBUG: Starting new HTTPS connection (1): timelapse-dev.azurewebsites.net:443                                                                                            
2022-07-02 08:00:59,056 DEBUG: https://timelapse-dev.azurewebsites.net:443 "POST /api/Image HTTP/1.1" 200 None                                                                                   
2022-07-02 08:00:59,063 DEBUG: Response code: 200                                                                                                                                                
2022-07-02 08:00:59,064 DEBUG: Image uploaded successfully                                                                                                                                       
2022-07-02 08:00:59,070 DEBUG: Response text:                                                                                                                                                    
2022-07-02 08:00:59,072 DEBUG: {                                                                                                                                                                 
    "id": 52937,                                                                                                                                                                                 
    "timestamp": "2022-07-01T20:00:53Z",                                                                                                                                                         
    "blobUri": "https://venaritimelapse.blob.core.windows.net/timelapse/11_2022-07-02_080053.jpg",                                                                                               
    "deviceId": 11,                                                                                                                                                                              
    "device": {                                                                                                                                                                                  
        "id": 11,                                                                                                                                                                                
        "serialNumber": "00000000638a42a2",                                                                                                                                                      
        "name": "Pi Zero W 2 number 3",                                                                                                                                                          
        "description": "3rd camera"                                                                                                                                                              
    }                                                                                                                                                                                            
}                                                                                                                                                                                                
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^
@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^
@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2022-07-03 13:42:03,463 INFO: Starting up uploadPendin
g.py...
ethandhunt commented 2 years ago

the pi could be running out of power while writing to files, this would explain the null bytes

ethandhunt commented 2 years ago

looks like that's what's happening

2022-07-02 08:00:56,297 DEBUG: {'batteryPercent': 2
ethandhunt commented 2 years ago

looks like skipping empty files has worked

ethandhunt commented 2 years ago

files that were empty:

$ file *
2022-07-02_080200.json: empty
2022-07-04_000054.json: empty
2022-07-04_080052.json: empty
2022-07-05_030054.json: empty
2022-07-05_050117.json: empty
2022-07-05_070105.json: empty
2022-07-05_143027.json: empty
2022-07-06_210054.json: empty
2022-07-06_220053.json: empty
2022-07-07_030058.json: empty
2022-07-07_070054.json: empty
2022-07-07_180104.json: empty
2022-07-07_190101.json: empty
2022-07-07_200054.json: empty
2022-07-07_230059.json: empty
2022-07-08_115021.json: empty
2022-07-09_180019.json: empty
2022-07-09_190100.json: empty
2022-07-09_200115.json: empty
2022-07-09_210054.json: empty
2022-07-10_010054.json: empty
2022-07-10_030101.json: empty
2022-07-10_040055.json: empty
2022-07-10_115654.json: empty
2022-07-10_180018.json: empty
2022-07-10_190111.json: empty
2022-07-10_210053.json: empty
2022-07-10_220052.json: empty
2022-07-10_230058.json: empty
2022-07-11_010105.json: empty
2022-07-11_020052.json: empty
2022-07-11_030054.json: empty
2022-07-11_050054.json: empty
2022-07-11_060053.json: empty
2022-07-11_111732.json: empty
2022-07-13_190105.json: empty
2022-07-13_200054.json: empty
2022-07-13_210054.json: empty
2022-07-13_220053.json: empty
2022-07-14_010053.json: empty
2022-07-14_020056.json: empty
2022-07-14_040054.json: empty
2022-07-14_070105.json: empty
2022-07-14_180011.json: empty
2022-07-14_220052.json: empty
2022-07-14_230054.json: empty
2022-07-15_010055.json: empty
2022-07-15_020052.json: empty
2022-07-15_030054.json: empty
2022-07-15_040104.json: empty
2022-07-15_050054.json: empty
2022-07-15_060054.json: empty
2022-07-15_140243.json: empty
2022-07-15_180021.json: empty
2022-07-15_190113.json: empty
2022-07-15_230054.json: empty
2022-07-16_000054.json: empty
2022-07-16_040053.json: empty
2022-07-16_060054.json: empty
2022-07-16_133853.json: empty
2022-07-16_180020.json: empty
2022-07-16_200054.json: empty
2022-07-16_210054.json: empty
2022-07-17_010054.json: empty
2022-07-17_020101.json: empty
2022-07-17_030054.json: empty
2022-07-17_040054.json: empty
2022-07-17_180011.json: empty
2022-07-17_190112.json: empty
2022-07-17_200052.json: empty
2022-07-17_210053.json: empty
2022-07-17_230054.json: empty
2022-07-18_000102.json: empty
2022-07-18_010053.json: empty
2022-07-18_050054.json: empty
2022-07-18_190105.json: empty
2022-07-18_210054.json: empty
2022-07-18_220055.json: empty
2022-07-19_030053.json: empty
2022-07-19_040053.json: empty
2022-07-19_060059.json: empty
2022-07-19_070054.json: empty
2022-07-20_180009.json: empty
2022-07-20_200059.json: empty
2022-07-21_000058.json: empty
2022-07-21_010054.json: empty
2022-07-21_030053.json: empty
2022-07-21_060057.json: empty
2022-07-21_070100.json: empty
2022-07-21_180022.json: empty
2022-07-21_190112.json: empty
2022-07-21_200053.json: empty
2022-07-21_220053.json: empty
2022-07-22_040054.json: empty
2022-07-22_114217.json: empty
2022-07-23_210054.json: empty
2022-07-23_220054.json: empty
2022-07-24_020053.json: empty
2022-07-24_030059.json: empty
2022-07-24_040100.json: empty
2022-07-24_060053.json: empty
2022-07-24_175105.json: empty
2022-07-24_180033.json: empty
2022-07-24_190107.json: empty
2022-07-24_200101.json: empty
2022-07-25_220058.json: empty
2022-07-25_230054.json: empty
2022-07-26_010052.json: empty
2022-07-26_040054.json: empty
2022-07-26_060054.json: empty
2022-07-28_222252.json: empty
2022-07-28_230053.json: empty
2022-07-29_050119.json: empty
2022-07-29_060116.json: empty
2022-07-29_070053.json: empty
2022-07-29_083236.json: empty