AiiDA extras

[giovannipizzi]

Currently all extras are accessible to users. If AiiDA has to store custom data, that we don't necessarily want the user to change without noticing, we don't currently have a namespacing scheme to allow this. Two current occurrences of this are for instance the hidden flag of a code and the hash of a Node (currently stored in extras but the user might inadvertently replace everything).

Proposed solution

• namespace AiiDA-reserved keys in extras (suggestion: prefix starts with _aiida_
• possibly leave this functionality available also to plugins, reserving _aiida_core_ to AiiDA-core, and suggesting plugins to use _aiida_entrypoint_ for plugins (e.g. _aiida_quantumespresso_).
• migrate current flags under this namespace where appropriate
• have get_extras(also_private=False) where the new also_private flag, False by default, establishes if key/value pairs starting with _aiida_ should be shown or not. (I chose also_private instead of, e.g., show_private because then the same name could be used also in the set methods, see below)
• TBD: if get_extra() should have the flag as well or not (my idea: not)
• adapt also other extra methods (e.g. set_extra(s) would complain with an exception when setting a key starting with _aiida_ unless a flag is set, reset_extras by default resets only extras without prefix, unless a flag is set)
  • also, ensure that this is the only current method to change extras, otherwise adapt also the other ones
• decide if the querybuilder should hide these results (probably not, to be documented - the other option is to specify a flag in the append of a node to a QueryBuilder object, but anyway this would apply only to projecting all extras, all other cases shouldn't have a flag I believe)
• As a bonus, in the tests for extras we wouldn't need (as we do now) to ignore all extras starting with _aiida_ (that is currently done because the hashing mechasism would add "unexpected" extras).

List of known internal extras

• Code.hidden: this is being used in import/export logic
• Node._aiida_hash
• Node._aiida_cached_from

[giovannipizzi]

Todo: discuss also what happens on export

[szoupanos]

This is not urgent but there should be a discussion on what should happen when extras are exported (and what is the behaviour of private extras). Issue #1761

[yakutovicha]

After today's discussion we agreed:

1. Instead of private extras call them aiida extras as it reflects better the idea of keeping them internally managed by aiida.
2. Create a separate api for the aiida extras and keep it hidden. 2.1. get_aiida_extra() / get_aiida_extras() 2.2. set_aiida_extra() / set_aiida_extra() 2.3. reset_aiida_extras() This will allow to avoid sometimes confusing interface due to the simultaneous managing of both types of extras.
3. For the moment keep both types of extras in the same table but later move the aiida extras to a separate one (maybe).
4. Implement a separate functionality to export the aiida extras.

[giovannipizzi]

I'm also not sure at this point if we should really call them aiida_extra. maybe metadata, or some other name (and document the difference between attributes, extras, metadata)