Uniform metadata in IM and Dashboard

EGI-ILM commented 1 year ago

This might be a difficult change to implement, but I think it could be very beneficial to store infrastructure metadata in the same way (or in the same location), regardless of the interface used (Dashboard, CLI, etc). As it is currently implemented, it seems that the Dashboard has an additional database for metadata, which is not used by the IM CLI.

For example, in the Dashboard it is very clear which infrastructures were created via the Dashboard and which ones via the CLI:

The Dashboard shows information about the site and VO of the infrastructures created from the Dashboard, but they are not shown in infrastructures created from the CLI (despite of site and VO being known).
From the Dashboard, the name can be manually changed for infrastructures created by IM CLI, but this is apparently not shown in the CLI, so it makes difficult to know from a first sight, which infrastructure is which:

$ im_client.py list
Infrastructure IDs: 
  d9f1cb12-71a2-11ed-99c3
  9b4ef014-6369-11ed-83f3
  86f8dd91-6469-11ed-bf85
  9f13a190-6167-11ed-a6e3
  75f15cb4-55d9-11ed-9a15
  6fa127de-546d-11ed-a27f
...

In the end, I always need to use both: I use the CLI to operate but use the Dashboard as assistant to match the ID with a name and know which infrastructure I am using (61a271de-72a2-0aed-11c3-1253dc2e5e77 is not friendly enough to identify an infrastructure from the CLI). A way to specify such metadata from the CLI would be useful.

There also appear to be some interoperability barriers. My im_client is using type = OpenStack but, nevertheless, infrastructures created from the CLI cannot be deleted from the Dashboard. This is probably why the status is "unknown" for the infrastructures created by the CLI. Maybe it is helpful to be able to operate on the infrastructures independently of the access credential configuration (as long as the permissions are OK, maybe it should not be relevant if the infrastructure was created with type = EGI or type = OpenStack, for example).
Outputs seem to be affected too.

micafer commented 1 year ago

Hi @EGI-ILM,

I will try to address some of these features separately. But I want first to address the interoperability issues that you are suffering, as it must not happen.

If you use the same OpenStack credentials in both applications, it must work in the same way in both.

We can try to debug this issue.

EGI-ILM commented 1 year ago

Thanks. I will contact you privately.

micafer commented 1 year ago

Hi @EGI-ILM,

We have implemented most of this functionality. You can test it in the IM devel instance.

In case that you create an infrastructure with the im-client tool, you can add a "infra_name" in the metadata of the TOSCA to set the name that will appear in the Dashboard.

tosca_definitions_version: tosca_simple_yaml_1_0

metadata:
  infra_name: some name

and you can use the filter option in the im-client to get the infrastructure with a particular name:

./im_client.py list "infra_name: some name"

Please test it in the IM devel instance, and let me know if it works as expected.

EGI-ILM commented 1 year ago

Site and VO and Operability comments above have not been addressed, so I only tested the name functionality:

IM CLI creates the name, and it is visible from the Dashboard

I created two infrastructures with name "lethe-datalake-1". They can be queried:


$ im_client.py list "infra_name: lethe-datalake-1"
Infrastructure IDs: 
359f9e64-7f87-11ed-a52b-ba0fc5d95973
e5674fdc-7f86-11ed-bed1-ba0fc5d95973

$ im_client.py list "infra_name: lethe-datalake" Infrastructure IDs: 359f9e64-7f87-11ed-a52b-ba0fc5d95973 e5674fdc-7f86-11ed-bed1-ba0fc5d95973

$ im_client.py list "infra_name: lethe" Infrastructure IDs: 359f9e64-7f87-11ed-a52b-ba0fc5d95973 e5674fdc-7f86-11ed-bed1-ba0fc5d95973

$ im_client.py list "infra_name: letheX" No Infrastructures.


**What does not seem to work or can be improved**
- The name is not shown in the IM CLI list, so it is still difficult to identify infrastructures:

$ im_client.py list Infrastructure IDs: 359f9e64-7f87-11ed-a52b-ba0fc5d95973 e5674fdc-7f86-11ed-bed1-ba0fc5d95973

- The CLI does not seem to provide renaming (just a comment, as I am not sure if this is desired, but the functionality exists in Dashboard).
- Using the Dashboard, I rename one of the infrastructures to "lethe-datalake-2". It seems that the CLI filter does not work anymore (do CLI and DSHB use a different data source for names?).

$ im_client.py list "infra_name: lethe-datalake-2" No Infrastructures. $ im_client.py list "infra_name: lethe-datalake-1" Infrastructure IDs: 359f9e64-7f87-11ed-a52b-ba0fc5d95973 e5674fdc-7f86-11ed-bed1-ba0fc5d95973


- Additionally, it _might_ be good to have a shortcut (e.g, -n "some name") for "infra_name: some name".

During the testing I also had an error I will send privately.

EGI-ILM commented 1 year ago

By the way, related to this naming functionality maybe it can be used also as a prefix for:

OpenStack VM
OpenStack volumes
OpenStack Security Groups

I know OpenStack VM names can be configured in a TOSCA template. I do not know how the other two could be renamed, but it is helpful to have this option, which makes resources easily identifiable without having to check the Infrastructure ID every time.

micafer commented 1 year ago

HI @EGI-ILM,

I have added new -n option in the im-client: https://imdocs.readthedocs.io/en/latest/client.html#cmdoption-im-client-n-name

For example you can do:

$ im_client.py getstate -n "lethe-datalake-1"

You can test it using the im-client master branch.

grycap / im

Uniform metadata in IM and Dashboard #1441