KenesisVideoFrameTransportCallback Error 4102

[chorhatarahuduketuri]

I'm taking part in an AWS-run community time trial race at work. I am only using the DeepRacer console, no custom SageMaker or ros anything.

I'm getting this error, and I have no idea why - is it related to trying to watch the video while the training is going on?

[s3] Successfully uploaded metrics to                  s3 bucket aws-deepracer-data-us-east-1-1 with s3 key data-e484e974-f9ba-4fae-8dbc-dc0d7b77b7f5/models/c399a855-842f-472c-bb8b-9481fb7848f5/metrics/training/training-20211215111804-lfOvfPcVQyi65y4LZ7waig.json.
Testing> Name=main_level/agent, Worker=0, Episode=20, Total reward=1.6403027683936851e+31, Steps=496, Training iteration=0
## agent: Finished evaluation phase. Success rate = 0.0, Avg Total Reward = 1.6403027683936851e+31
Waiting to flush the Mp4 queue for racecar_0...
Done flushing the Mp4 queue for racecar_0...
[s3] Successfully uploaded simtrace_training to              s3 bucket aws-deepracer-data-us-east-1-1 with s3 key data-e484e974-f9ba-4fae-8dbc-dc0d7b77b7f5/models/c399a855-842f-472c-bb8b-9481fb7848f5/sim-trace/training/training-simtrace/0-iteration.csv.
[s3] Successfully uploaded pip to              s3 bucket aws-deepracer-data-us-east-1-1 with s3 key data-e484e974-f9ba-4fae-8dbc-dc0d7b77b7f5/models/c399a855-842f-472c-bb8b-9481fb7848f5/videos/training/training-20211215111804-lfOvfPcVQyi65y4LZ7waig/camera-pip/0-video.mp4.
[s3] Successfully uploaded degree45 to              s3 bucket aws-deepracer-data-us-east-1-1 with s3 key data-e484e974-f9ba-4fae-8dbc-dc0d7b77b7f5/models/c399a855-842f-472c-bb8b-9481fb7848f5/videos/training/training-20211215111804-lfOvfPcVQyi65y4LZ7waig/camera-45degree/0-video.mp4.
[s3] Successfully uploaded topview to              s3 bucket aws-deepracer-data-us-east-1-1 with s3 key data-e484e974-f9ba-4fae-8dbc-dc0d7b77b7f5/models/c399a855-842f-472c-bb8b-9481fb7848f5/videos/training/training-20211215111804-lfOvfPcVQyi65y4LZ7waig/camera-topview/0-video.mp4.
[ WARN] [1639567602.880610919, 69.944000000]: [KinesisVideoFrameTransportCallback] dr-kvs-372026249783-20211215111803-e59dae08-20a0-43e4-9f74-b0bd568d503c PutFrame failed. Error code: 4102

[chorhatarahuduketuri]

All my models are now returning this about 10 minutes into training.

[chorhatarahuduketuri]

Unknown if related, but now this is happening:

================================================================================REQUIRED process [master] has died!
process has died [pid 206, exit code -15, cmd rosmaster --core -p 11311 -w 3 __log:=/root/.ros/log/cacc20c8-5da2-11ec-a780-0e69a02937b3/master.log].
log file: /root/.ros/log/cacc20c8-5da2-11ec-a780-0e69a02937b3/master*.log
Initiating shutdown!
================================================================================
[agent-9] killing on exit
[agents_video_editor-8] killing on exit
[car_reset_node-7] killing on exit
[gazebo-6] killing on exit
[racecar/kinesis_video_streamer-5] killing on exit
[racecar/h264_video_encoder-4] killing on exit
[racecar/robot_state_publisher-3] killing on exit
[racecar/controller_manager-2] killing on exit
[INFO] [1639571905.383233, 279.539000]: Shutting down spawner. Stopping and unloading controllers...
[download_params_and_roslaunch_agent_node-2] killing on exit
[rosout-1] killing on exit
[INFO] [1639571906.404158, 279.539000]: Stopping all controllers...
[libx264 @ 0x55b579c6fb00] frame I:464   Avg QP:12.31  size: 71548
[libx264 @ 0x55b579c6fb00] frame P:6441  Avg QP:15.19  size: 12546
[libx264 @ 0x55b579c6fb00] mb I  I16..4: 16.7% 11.6% 71.6%
[libx264 @ 0x55b579c6fb00] mb P  I16..4:  1.7%  1.3%  1.4%  P16..4: 33.2% 16.8% 10.8%  0.0%  0.0%    skip:34.8%
[libx264 @ 0x55b579c6fb00] final ratefactor: 16.93
[libx264 @ 0x55b579c6fb00] 8x8 transform intra:18.6% inter:29.3%
[libx264 @ 0x55b579c6fb00] coded y,uvDC,uvAC intra: 82.4% 69.8% 58.5% inter: 33.8% 14.0% 6.3%
[libx264 @ 0x55b579c6fb00] i16 v,h,dc,p: 20% 60%  9% 11%
[libx264 @ 0x55b579c6fb00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  3% 37% 31%  3%  4%  2%  8%  2% 11%
[libx264 @ 0x55b579c6fb00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  6% 28% 17%  6%  7%  4% 13%  4% 14%
[libx264 @ 0x55b579c6fb00] i8c dc,h,v,p: 51% 38%  7%  4%
[libx264 @ 0x55b579c6fb00] Weighted P-Frames: Y:0.1% UV:0.0%
[libx264 @ 0x55b579c6fb00] kb/s:1981.31
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
[WARN] [1639571907.427191, 279.539000]: Controller Spawner error while taking down controllers: unable to contact master