Analytics fails with KeyError: 'tree'

[SrinivasMushnoori]

I'm using this in an EnTK context, and the stack is:

  python               : 2.7.14
  pythonpath           : 
  virtualenv           : /home/scm177/VirtualEnvs/Env_RepEx

  radical.analytics    : v0.45.2-104-ge615742@feature-entk-integration
  radical.entk         : 0.7.8-0.7.8@detached-0.7.8
  radical.pilot        : 0.50.13-v0.50.13-3-ga3a15b2@devel
  radical.utils        : 0.50.1-v0.50.1-2-gfebbcd0@devel
  saga                 : 0.50.0-v0.50.0@devel

The error is below:

Traceback (most recent call last):
  File "/home/scm177/VirtualEnvs/Env_RepEx/bin/repex", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/scm177/RADICAL-LAB/repex_3/bin/repex", line 124, in <module>
    src   = pwd)
  File "/home/scm177/VirtualEnvs/Env_RepEx/local/lib/python2.7/site-packages/radical/analytics/session.py", line 111, in __init__
    self._initialize_entities(self._profile)
  File "/home/scm177/VirtualEnvs/Env_RepEx/local/lib/python2.7/site-packages/radical/analytics/session.py", line 215, in _initialize_entities
    details = self._description['tree'].get(uid, dict())
KeyError: 'tree'

The EnTK (and analytics) script is here . Logs are here

Thanks. Happy to assist with debugging.

[SrinivasMushnoori]

@andre-merzky @vivek-bala has there been movement on this?

[andre-merzky]

This looks like an entk session. Pinging @vivek-bala - do you have an idea whats up with that session?

[vivek-bala]

Yea, I came across this with Giannis. For a quick fix, you probably have a file called 'radical.entk..json' which you can rename to '.json' (make sure to backup any other files with the same latter name). You can rerun analytics again.

I'll be making a release today and this issue should be resolved in it. You won't have to make any filename changes.

[SrinivasMushnoori]

@vivek-bala has this been addressed? I still see this error with EnTK 0.7.10 devel.

[vivek-bala]

Hey Srinivas, this should be addressed in the latest release. Devel as well. But I recommend using the released version

[SrinivasMushnoori]

Just tried using master, I still see this:

Traceback (most recent call last):
  File "/home/scm177/VirtualEnvs/Env_RepEx/bin/repex", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/scm177/RADICAL-LAB/repex_3/bin/repex", line 125, in <module>
    src   = pwd)
  File "/home/scm177/VirtualEnvs/Env_RepEx/local/lib/python2.7/site-packages/radical/analytics/session.py", line 92, in __init__
    self._description = re.utils.get_session_description(sid=sid, src=self._src)
  File "/home/scm177/VirtualEnvs/Env_RepEx/local/lib/python2.7/site-packages/radical/entk/utils/prof_utils.py", line 217, in get_session_description
    desc = ru.read_json("%s/%s/radical.entk.%s.json" % (src, sid, sid))
  File "/home/scm177/VirtualEnvs/Env_RepEx/local/lib/python2.7/site-packages/radical/utils/read_json.py", line 26, in read_json
    with open (filename) as f:
IOError: [Errno 2] No such file or directory: '/home/scm177/RADICAL-LAB/repex.experiments/experiments/gromacs_runs/./re.session.mcewan.engr.rutgers.edu.scm177.017854.0019/radical.entk../re.session.mcewan.engr.rutgers.edu.scm177.017854.0019.json'

[SrinivasMushnoori]

Not the same error but definitely something to do with the json file being named incorrectly...

[vivek-bala]

I think the expected mode is as follows:

• You should have a json file from EnTK with the name format 'radical.entk..json'.
• Profiles should be contained in a folder. Name of the folder is the sid
• src should point to the location of this folder

Can you check if any of these conditions are not met/altered?

[SrinivasMushnoori]

The name of the json is:

re.session.mcewan.engr.rutgers.edu.scm177.017854.0019.json

It is NOT of the form radical.entk..json

The other two are met.

[vivek-bala]

Okk. Thanks for checking. Was the JSON created with 0.7.10? If not, do you have an idea about the version you used that created the json file?

[SrinivasMushnoori]

This is 0.7.10 master. I am using an analytics script that is tied to my EnTK script. It is executed as soon as the workflow is complete.

[SrinivasMushnoori]

If it helps, here is the EnTK + RA script: https://github.com/SrinivasMushnoori/repex/blob/usecase/peptide_gromacs/bin/repex#L122

[vivek-bala]

Hmmm. I see that in 0.7.10, the json is created with the initial 'radical.entk' here. Let me know if you see this error again.

FWIW, it tells me that I should also mention the version in the json file.

[SrinivasMushnoori]

I just retried. I can reproduce this.

[SrinivasMushnoori]

Update: it's looking for this file: radical.entk../re.session.mcewan.engr.rutgers.edu.scm177.017854.0020.json The file that is present is: radical.entk.re.session.mcewan.engr.rutgers.edu.scm177.017854.0020.json

There seems to be a stray ./ in the file name that appears to be triggering the error.

[SrinivasMushnoori]

Fixed it. The reason it was happening was that there was a ./ in the RA script I was using (presumably because it was required in an older version?) Removing that part makes it work as intended. Closing now, reopen as needed. Thanks.