Missing events

[jdakka]

*Stack

radical.analytics    : v0.45.2-104-ge615742@feature-entk-integration
  radical.entk         : 0.7.6-0.7.6@master
  radical.pilot        : 0.50.9-v0.50.9-9-ga4ae131d@master
  radical.utils        : 0.50.2-v0.50.2@master
  saga                 : 0.50.0-v0.50.0@master

• notebook to follow along

• Input:

rp_units = rp_session.filter(etype='unit', inplace=False)
durations=list()
exec_duration = rp_units.duration(state=[rp.AGENT_EXECUTING,
                                    rp.AGENT_STAGING_OUTPUT_PENDING])

start_data = rp_units.timestamps(event = [{ru.EVENT: 'exec_start'}])
stop_data = rp_units.timestamps(event = [{ru.EVENT: 'exec_stop'}])

print exec_duration
print start_data
print stop_data

• Output:

['session_close', 'config_parser_stop', 'staging_in_start', 'session_start', 'staging_in_stop', 'get', 'component_init', 'drop', 'submission_start', 'session_stop', 'submission_stop', 'state', 'update_request', 'update_pushed', 'flush', 'put', 'sync_abs', 'config_parser_start', 'component_final', 'setup_done', 'END']
0
[]
[]

ERR from verbose:

((Permission denied (gssapi-keyex,gssapi-with-mic).
)) (/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/utils/pty_exceptions.py +61 (translate_exception)  :  e = se.PermissionDenied (cmsg))
2018-10-03 04:24:11,126: radical.utils       : MainProcess                     : MainThread     : ERROR   : Could not create lease object
Traceback (most recent call last):
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/radical/utils/lease_manager.py", line 175, in _create_object
    obj = _LeaseObject (self, self._log, creator, args)
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/radical/utils/lease_manager.py", line 33, in __init__
    self.obj        = creator (*args)
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py", line 284, in _shell_creator
    return sups.PTYShell(url, self.get_session(), self._logger)
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 248, in __init__
    self.pty_shell  = self.factory.run_shell  (self.pty_info)
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 450, in run_shell
    self._initialize_pty (sh_slave, info)
  File "/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 413, in _initialize_pty
    raise ptye.translate_exception (e)
PermissionDenied: read from process failed '[Errno 5] Input/output error' : (Permission denied (gssapi-keyex,gssapi-with-mic).
) ((Permission denied (gssapi-keyex,gssapi-with-mic).
)) (/home/jdakka/hyperspace/python-2.7-ve/local/lib/python2.7/site-packages/saga/utils/pty_exceptions.py +61 (translate_exception)  :  e = se.PermissionDenied (cmsg))

• and from client logs:

rp.session.two.jdakka.017806.0005.log:2018-10-03 04:24:09,379: rp.session.two.jdakka.017806.0005: MainProcess                     : MainThread     : ERROR   : failed to fet profile for pilot.0000

• Before execution, I ran myproxy with a higher time quanta: myproxy-logon -s myproxy.xsede.org -l jdakka -t 100

• grid-proxy-info showed that my certificate was still valid

• I ran radical-pilot-fetch-profiles, which did not produce a gsissh error, but even after running this command I still don't have all the events.

• I can see the events (such as cu_exec_start/cu_exec_stop) in the agent logs: pilot.0000.zip

[andre-merzky]

Hi Jumana,

can you please attach a tarball of all profiles and the json file, so that I can try to reproduce this?

Thanks!

(PS: please name compressed tarfiles .tgz or .tar.gz - .zip makes the impression that the files can be unzipped - but that fails, and one has to guess or inspect the file type...)

[jdakka]

Github doesn't accept attachments with .tgz extensions so I've compressed to .tar.gz:

rp.session.two.jdakka.017806.0005.tar.gz

pilot.0000.tar.gz