Shared Python lib

[c0c0n3]

Summary

SUPSI, CSIC, Rovimatica, Ginkgo and Martel could all potentially benefit from a Python library for interacting with the Kitt4sme infra, especially Keycloak and Fiware. Developing a common code base and sharing it as a library is probably the way to go.

Intended outcome

Figure out if we should roll out our own thingie (e.g. FiPy) or rather entirely rely on existing libs like PyNGSI and PyNGSI-LD. We should come up w/ a solution that provides:

• Type-annotated records to easily define NGSI v2 and LD entities
• Built-in validation of NGSI entities
• Orion and QuantumLeap clients
• Transparent handling of Fiware headers---setting or propagation of tenant, service path and correlation ID.
• Transparent handling of Keycloak security---JWT token acquisition, propagation and refresh.

How will it work?

Kitt4sme Python services would simply import the library and take advantage of all the built-in functionality. E.g. see how Roughnator uses FiPy.

[vcutrona]

We should consider also python-orion-client in our analysis. All the considered libs are from the same organization (Orange-Open Source).

[vcutrona]

PyNGSI

Reqs

• [ ] Type-annotated records to easily define
  • [x] NGSI v2 entities
  • [ ] NGSI-LD entities
• [ ] Built-in validation of NGSI entities
• [ ] Clients
  • [x] Orion
  • [ ] QuantumLeap
• [ ] Transparent handling of Fiware headers---setting or propagation of
  • [x] tenant,
  • [x] service path
  • [ ] correlation ID.
• [ ] Transparent handling of Keycloak security---JWT token
  • [ ] acquisition
  • [ ] propagation
  • [ ] refresh.

Comments

• Not maintained (last commit 1y ago)
• Useful for developing "bridge" agents (converting "rows" from Sources to NGSI entities), with some ready-to-use implementations for MQTT, FTP, and generic HTTP server.
• Tenant and auth configs managed via config file
• Light entity validation (forbidden chars for id and other fields, reserved keywords)

[vcutrona]

python-ngsild-client

Reqs

• [ ] Type-annotated records to easily define
  • [ ] NGSI v2 entities
  • [x] NGSI-LD entities
• [ ] Built-in validation of NGSI entities
• [ ] Clients
  • [x] Orion
  • [ ] QuantumLeap
• [ ] Transparent handling of Fiware headers---setting or propagation of
  • [x] tenant,
  • [ ] service path
  • [ ] correlation ID.
• [ ] Transparent handling of Keycloak security---JWT token
  • [ ] acquisition
  • [ ] propagation
  • [ ] refresh.

Comments

• Under development (many contributions pushed in the last month)
• Revolving around the concept of agents (converting "rows" from Sources to NGSI entities).
• Light entity validation (fields presence - no actual type checking)

[vcutrona]

PyNGSI-LD

Reqs

• [ ] Type-annotated records to easily define
  • [ ] NGSI v2 entities
  • [x] NGSI-LD entities
• [ ] Built-in validation of NGSI entities
• [ ] Clients
  • [x] Orion
  • [ ] QuantumLeap
• [ ] Transparent handling of Fiware headers---setting or propagation of
  • [ ] tenant,
  • [ ] service path
  • [ ] correlation ID.
• [ ] Transparent handling of Keycloak security---JWT token
  • [ ] acquisition
  • [ ] propagation
  • [ ] refresh.

Comments

• Under development (project started in January 2022)
• Revolving around the concept of agents (converting "rows" from Sources to NGSI entities).
• Entity creation and validation delegated to python-ngsild-client

[vcutrona]

Given the above comments, my view is that rolling out FiPy is the best choice for us, because:

• Security stuff is never managed in considered libs, while FiPy already supports some aspects
• Other libs are mainly focused on mapping different sources (CSV, JSON, HTTP Server, MQTT messages, etc.), while we are also interested in managing different outputs (Orion and QuantumLeap), by also considering the security (mostly overlooked in the other libs, but crucial for KITT4SME, as well as for production envs)
• Validation is never considered at the type checking level - FiPy models are validated with pydantic

Alternatively, we may consider extending a lib (pyngsild?) to meet our requirements, but:

• NGSI v2 and LD versions are supported by different libs
• Libs for NGSI-LD are quite new and not stable, thus we should deal with their current evolving status
• Most of our requirements are already implemented in FiPy, thus we will end up migrating Fipy functions to an external lib

Instead, I would consider bringing some features (e.g., fields presence check, reserved keywords check, agents creation, sources implementation) from the considered libs to FiPy (considered libs are released under Apache2 license).

[c0c0n3]

Wow! Excellent analysis, thanks a stack!! Let's make a decision about this in the next grooming session.