search

ging / fiware-draco

The Draco Generic Enabler is an alternative data persistence mechanism for managing the history of context. It is based on Apache NiFi and is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic and also offers an intuitive graphical interface
https://fiware-draco.readthedocs.io/en/latest/
Apache License 2.0
20 stars 14 forks source link

Column-Like Storing #31

Closed ianp1 closed 3 years ago

ianp1 commented 3 years ago

Greetings,

from the Draco Documentation over at readthedocs it states that Column-Like storing is available for NGSIToMysql-Processor. However, when running draco via Docker in latest version, the "Attribute Persistence"-Option "Column" ist not available. From the code you can see, that the option "column" is not added to the allowable values:

https://github.com/ging/fiware-draco/blob/adcefbd233702d920fba2c5f2e118161b13be975/nifi-ngsi-bundle/nifi-ngsi-processors/src/main/java/org/apache/nifi/processors/ngsi/NGSIToMySQL.java#L80

Is this intentional? If this feature is not yet implemented, shouldn't that be stated in the documentation?

Kind regards

anmunoz commented 3 years ago

Hi,

Yes, you are right, this option is not available for that processor yet, Sorry for the error in the documentation.

ianp1 commented 3 years ago

No worries. If you could give me a short summary on what steps are necessary, i might have a look at it once i have some spare time.

anmunoz commented 3 years ago

Thanks, You can enable this option in the processor in the same line that you show adding the column option .allowableValues("row","column") Aso you need to code the column option in this file:

https://github.com/ging/fiware-draco/blob/adcefbd233702d920fba2c5f2e118161b13be975/nifi-ngsi-bundle/nifi-ngsi-processors/src/main/java/org/apache/nifi/processors/ngsi/ngsi/backends/MySQLBackend.java#L24

And also build the strings for making the queries according to the documentation.

If you need some additional information please let me know

ianp1 commented 3 years ago

Hey again, so i did look into the files you pointed to. I followed along to the implementation in PostgreSQLBackend and it did seem to work. You can check it out in my fork over here. However, i am a little concerned about changes in entities. Since the columns in sql tables are static, new attributes would cause an error and would fail. Any thoughts on that?

anmunoz commented 3 years ago

Hello, as you mention if you receive new attributes in a database or table that have been already created, the database will generate an error. For that reason, we show in the documentation that you have to be careful with that and, one solution is changing the fiware-service or fiware-service path in order that a new table will be created.

Hence, your code seems good to me, if you want you can contribute that creating a pull request.