josephjclark
commented
1 month ago We also have to append the collection adaptor to every workflow.
I would really like to use a heuristic to decide whether to do this, because it's strange to see the collection adaptor being loaded in logging. We should be able to tell with a simple scan of the expressions whether collections. is being called, and then just harmlessly load it
Later, Lightning itself should make the decision to include a collection adaptor based on some user input
The worker needs to take the run token JWT and write it to the credential of every run. It also needs to add the collections adaptor to every step (probably we do this bit as part of #777 actually - and we can only do it when we have a working adaptor (a stub one would be fine))
It should write the token as
collections_keyJob code will see the key and credential, but it's harmless and safe.
Maybe later we'll only append the credential for jobs which we know use the collections API. But I think to start with we'll switch it on for everyone by default.
Note that the run token JWT is totally safe and secure. It doesn't matter if users or attackers get hold of it:
The absolute worst case, from a security point of view, is that an attacker gets hold of the worker token and is able to connect to the socket. Then, while the token is unexpired and the run is active, they can only access credentials and dataclips related to that project. Frankly, if you want a credential related to that project, there are far easier ways to steal it.