Plumbum fails to encode output of make command

nathanlcarlson commented 5 years ago

make[2]: Entering directory `/usr/local/esgf-dashboard/src/c/esgf-dashboard-ip/src'
gcc -DHAVE_CONFIG_H -I. -I..    -O2 -I/usr/include -g -O2 -I. -I../include  `/usr/local/conda/envs/esgf-pub/bin/xml2-config --cflags` -I/usr/local/geoip/include -DGEOIP_DATA_PATH=\"/usr/local/geoip\" -DALLOW_FEDERATION=\"no\"  -g -O2 -MT esgf_dashboard_ip-esgf-dashboard-ip.o -MD -MP -MF .deps/esgf_dashboard_ip-esgf-dashboard-ip.Tpo -c -o esgf_dashboard_ip-esgf-dashboard-ip.o `test -f 'esgf-dashboard-ip.c' || echo './'`esgf-dashboard-ip.c
Traceback (most recent call last):
  File "esg_node.py", line 437, in <module>
    main()
  File "esg_node.py", line 320, in main
    system_component_installation(esg_dist_url, node_type_list)
  File "esg_node.py", line 183, in system_component_installation
    esg_dashboard.main()
  File "/home/jenkins/repos/esgf-installer/data_node/esg_dashboard.py", line 161, in main
    setup_dashboard()
  File "/home/jenkins/repos/esgf-installer/data_node/esg_dashboard.py", line 73, in setup_dashboard
    run_dashboard_script()
  File "/home/jenkins/repos/esgf-installer/data_node/esg_dashboard.py", line 157, in run_dashboard_script
    esg_functions.call_binary("make")
  File "/home/jenkins/repos/esgf-installer/esgf_utilities/esg_functions.py", line 951, in call_binary
    output = command.run_tee()
  File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/plumbum/commands/base.py", line 271, in run_tee
    return self._use_modifier(plumbum.commands.modifiers.TEE, kwargs)
  File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/plumbum/commands/base.py", line 250, in _use_modifier
    return self & modifier_instance
  File "/usr/local/conda/envs/esgf-pub/lib/python2.7/site-packages/plumbum/commands/modifiers.py", line 207, in __rand__
    tee_to[fd].write(text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 33: ordinal not in range(128)

This did not arise in manual testing. The first solution that comes to mind is to be able to silence output and maybe this won't happen. Otherwise this UnicodeEncodeError could be caught and handled, if it doesn't interrupt/stop the underlying process. @William-Hill Thoughts?

William-Hill commented 5 years ago

@nathanlcarlson I think catching the exception and logging it (probably at the warning level) could be a good initial test. Checking the return code may let us know if the underlying process completed

nathanlcarlson commented 5 years ago

@William-Hill That sounds good, I'll get started on it.

nathanlcarlson commented 5 years ago

@William-Hill I was thinking we could simply re-run the command silently to ensure the process succeeds.

William-Hill commented 5 years ago

@nathanlcarlson That works too. I think using the BG modifier in plumbum with still throw an exception if something in the process goes wrong so we could maintain integrity

ESGF / esgf-installer

Plumbum fails to encode output of make command #570