Can't download custom_files from s3_bucket

[decarv]

Have been running into this problem when trying to start the simulation on OSX. For some reason I cant download the custom_files from the s3 bucket. Any clues?

process[agent-9]: started with pid [1220]
+ export PYTHONUNBUFFERED=1
+ PYTHONUNBUFFERED=1
+ python3 -m markov.rollout_worker
Initializing SageS3Client...
S3 bucket: bucket
S3 prefix: rl-deepracer-sagemaker
{"simapp_exception": {"version": "1.0", "date": "2019-08-14 15:29:42.100910", "function": "download_file", "message": "Exception [Could not connect to the endpoint URL: \"http://172.18.0.1:9000/bucket/custom_files/model_metadata.json\"] occcured on download file-./custom_files/model_metadata.json from s3 bucket-bucket key-custom_files/model_metadata.json", "errorCode": "401", "exceptionType": "s3_datastore.exceptions", "eventType": "user_error"}}
Could not download custom model metadata from custom_files/model_metadata.json, using defaults.
Loaded default action space.
{"simapp_exception": {"version": "1.0", "date": "2019-08-14 15:29:49.831050", "function": "download_file", "message": "Exception [Could not connect to the endpoint URL: \"http://172.18.0.1:9000/bucket/custom_files/reward.py\"] occcured on download file-./custom_files/customer_reward_function.py from s3 bucket-bucket key-custom_files/reward.py", "errorCode": "401", "exceptionType": "s3_datastore.exceptions", "eventType": "user_error"}}
{"simapp_exception": {"version": "1.0", "date": "2019-08-14 15:29:49.840151", "function": "download_customer_reward_function", "message": "Could not download the customer reward function file. Job failed!", "errorCode": "503", "exceptionType": "simulation_worker.exceptions", "eventType": "system_error"}}
NoneType
[WARN] [1565796590.379974, 0.000000]: Controller Spawner couldn't find the expected controller_manager ROS interface.
================================================================================REQUIRED process [agent-9] has died!
process has died [pid 1220, exit code 1, cmd /app/robomaker-deepracer/simulation_ws/install/deepracer_simulation/lib/deepracer_simulation/run_rollout_rl_agent.sh __name:=agent __log:=/root/.ros/log/3db0388c-bea8-11e9-8048-0242ac120003/agent-9.log].
log file: /root/.ros/log/3db0388c-bea8-11e9-8048-0242ac120003/agent-9*.log
Initiating shutdown!
================================================================================
[agent-9] killing on exit
[better_odom-8] killing on exit
[car_reset_node-7] killing on exit
[robot_state_publisher-6] killing on exit
[racecar/controller_manager-5] killing on exit
[racecar_spawn-4] killing on exit
[gazebo_gui-3] killing on exit
[gazebo-2] killing on exit
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/gazebo_ros/spawn_model", line 313, in <module>
    sm.callSpawnService()
  File "/opt/ros/kinetic/lib/gazebo_ros/spawn_model", line 271, in callSpawnService
    initial_pose, self.reference_frame, self.gazebo_namespace)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/gazebo_ros/gazebo_interface.py", line 28, in spawn_urdf_model_client
    rospy.wait_for_service(gazebo_namespace+'/spawn_urdf_model')
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 159, in wait_for_service
    raise ROSInterruptException("rospy shutdown")
rospy.exceptions.ROSInterruptException: rospy shutdown
Unhandled exception in thread started by 
sys.excepthook is missing
lost sys.stderr
Traceback (most recent call last):
  File "/app/robomaker-deepracer/simulation_ws/install/deepracer_simulation/lib/deepracer_simulation/car_node.py", line 133, in <module>
    DEEPRACER = DeepRacer()
  File "/app/robomaker-deepracer/simulation_ws/install/deepracer_simulation/lib/deepracer_simulation/car_node.py", line 44, in __init__
    rospy.wait_for_service('/gazebo/set_model_state')
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 159, in wait_for_service
    raise ROSInterruptException("rospy shutdown")
rospy.exceptions.ROSInterruptException: rospy shutdown
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

[joezen777]

What does your minio startup output look like? Can you access minio from that 172 URL? Has this worked before?

[decarv]

I couldn't fix this problem, so I just uninstalled and reinstalled everything and it worked just fine. I'm closing this issue.

[isaacmtz90]

I couldn't fix this problem, so I just uninstalled and reinstalled everything and it worked just fine. I'm closing this issue.

I managed to fix the issue by pruning my containers and re-downloading the robomaker and sagemaker images.