resilience to external downloads

[marksantcroos]

I regularly get the error below, we might want to reduce having these external dependencies as much as possible.

Error: Couln't open/download resource configuration file 'https://raw.github.com/radical-cybertools/radical.pilot/master/configs/xsede.json': HTTP Error 503: backend read error.

[andre-merzky]

Agree - I think the default should be to use the configs which got installed with RP ...

[andre-merzky]

oops, didn't mean to close...

[oleweidner]

+1 I'll try to implement that! Generally, I would like to overhaul the configuration system a bit as it is currently a bit clunky. I propose the following changes:

• Currently you can only pass filenames as resource configurations. This is clunky, especially if you want to try out something quick. It would make sense to allow to pass the data structure itself.
• Data structure is currently a dict(). I would like to make it a proper class (maybe inheriting from attribute interface)?
• Currently, configurations need to be passed "explicitly". I think RP should load a set of "default" configurations automatically if nothing else is provided.
• "local" submission (e.g., pbs://localhost) shouldn't be an extra entry, but should rather be detected automatically.

Most changes would be minimally invasive and would bring a great deal of convenience. I would like to tackle them all at once if possible.

Thoughts?

[oleweidner]

Can I have some feedback on this please? ;-)

[andre-merzky]

I was hesitating with comments as I am so biased on the issue :/ Anyways, here it goes, please take it with a grain of salt, or two ;)

Currently you can only pass filenames as resource configurations. This is clunky, especially if you want to try out something quick. It would make sense to allow to pass the data structure itself.

The Troy session constructor accepts:

• strings/URLs: are interpreted as locations of config files. If they point to directories, all files therein are used. Config files are in JSON (with python style comments filtered out)
• dicts: those are interpreted just as parsed JSON configs

Default configs are looked for in $HOME/.troy/ and in $TROY_CONFIG_PATH (or something like that), and whatever is passed to the session as per above. Later configs overwrite newer configs, and configs can reference each other.

This would likely be an overkill for RP -- the point is that using JSON, and passing either location strings or dicts to the RP session, would be nicely in sync with Troy...

Data structure is currently a dict(). I would like to make it a proper class (maybe inheriting from attribute interface)?

In principle I would agree, for type safety etc. Also, other than Troy, RP has a rather fixed set of settings which is unlikely to be extended by user or backend specific keywords, so that would work well. OTOH, type checks and conversions are easily done on the fly.

Currently, configurations need to be passed "explicitly". I think RP should load a set of "default" configurations automatically if nothing else is provided.

Absolutely. And those default configs should be part of the installation, and not be spread over the code.

"local" submission (e.g., pbs://localhost) shouldn't be an extra entry, but should rather be detected automatically.

it should be the default I guess :) I am not sure though you can reliably detect all local config settings for, say, stampede or archer...

Cheers, Andre.

[oweidner]

Hi Andre, thanks for the feedback. I think we're pretty much on the same page... One think that comes to mind immediately when you compare RP and TROY configuration: RP is never meant to be seen or touched by the user, unless a new machine / cluster is added. This is distinctively different from TROY, where (as far as I understand) each run requires adjustment of the configuration files, i.e., the users deal with those actively as part of the overall workload execution process.

I will start implementing something, for now sans $HOME/. configuration paths. Obviously, this can easily be added at a later stage without messing with anyones code. This is one of the things that I would like to pump out before August 1st while we still have the flexibility.

[marksantcroos]

I dont have too many requirements for the configuration. For one thing, I consider it much more static than troy configuration, and mainly add-once-never-touch again.

There is one thing I noticed though, the task_launch_method disappeared from the futuregrid.json. Was that on purpose is accidental?

Related, this is also the only value that I expect to be changing on a per-CU basis regularly. But I guess I wont do this through this interface, but explicitly in the CU description.

Gr,

Mark

[andre-merzky]

Agreements all around, I like that :)

[marksantcroos]

Changes made after PM start-up don't seem to propagate. My explanation would be that the PM gives a copy of the config to the workers pretty early and they will never see the changes.

If maintaining a synchronized version is hard, then an easier workaround would probably be to do this magic on the session as you can do that before the PM starts or have it as an independent element altogether and then pass it as an argument to the PM instantiation.

[andre-merzky]

Changes made to what? I don't think that a pilot description should change after pilot startup? Probably missing something...

[marksantcroos]

I was referring to the commit that resolved this issue (65d4abb).

[marksantcroos]

Main topic solved, created new ticket for config changing #163.