Cope with not id compliant information in the NGSIv2 retrieve operations

[fgalan]

Edited issue text (30/11/2016):

Orion restrict id fields (in NGSIv2 and NGSIv1 with -strictNgsiv1Ids) at the input requests. However, it could happen that some field exist in DB with an invalid id due to:

• It came from an older Orion version,
• Orion is running without -strictNgsiv1Ids and the entity was created using NGISv1 but retrieved using NGSIv2

The are 3 strategies to cope with this situation:

• Return the no-NGSIv2-compliant entities but appliying some "neutralization strategy" in order not violate the id rules. This is similar to the procedure to neutralize forbidden characters ("<", etc.) already in place. In particular, the following neutralizations could be applied:
  • id fields exceding length will be truncated to maximum length
  • id fields of zero length will be replaced by "none"
  • id-forbidden chars (e.g. whitespaces) will be replaced by "_"
• Remove the elements violating NGSIv2 id rules. In particular:
  • if a metadata name or type violates the rules, the metadata is not returned as part of the attribute metadata
  • if an attribute name of type violates the rules, the attribute is not returned as part of the entity attributes
  • if an entity id or type violates the rules, the entity is not returned
• Not return any entity which violate the rules in some point (either in the entity id and type themselves or in some contained attribute name/type and/or metadata name/type).

---

Original issue text:

(Spin-off of #1601)

Orion restrict id fields (in NGSIv2 and NGSIv1 with -strictNgsiv1Ids) at the input requests. However, it could happen that some field exist in DB with an invalid id due to it came from an older Orion version, Orion is running without -strictNgsiv1Ids and the entity was created using NGISv1 but retrieved using NGSIv2, etc.

Thus, we need to apply some "neutralization" strategy in NGSIv2 and NGSIv1 with -strictNgsiv1Ids for these cases. In particular:

• id fields exceding length will be truncated to maximum length
• forbidden chars in id fields will be replaced by "_"

Currently, we are applying a similar procedure to neutralize forbidden characters ("<", etc.) but I'm not sure if it is directly applicable here (that procedure process all the output, while here we have to be selective and neutralize only on id fields).

Efort: 2 man day

[fgalan]

Additional "rule", taking into account https://github.com/telefonicaid/fiware-orion/issues/1836:

• id fields of zero length will be replaced by "none"

(In some sense, PR https://github.com/telefonicaid/fiware-orion/pull/1835 is addresing that for entity/attribute/metadata types, but entity id and attribute/metadata name should be also taken into account).

[kzangeli]

It seems to me that what we need here is a migration script, not part of the broker implementation.

If done inside the broker, imagine we have an entity with Id you&me in mongo (& is a forbidden char, right?). We find it and returns it with id you_me but in mongo it keeps having the id you&me ... I find this pretty strange. I think the db should be migrated and no change be made to the broker. Probably not possible, but if the GET is done with you&me and we return an entity with you_me ... that would be almost crazy ...

[fgalan]

It cannot be solved with a migration script as the problem can happen in runtime. Consider the following example:

• User creates an entity using NGSIv1 which id is "you&me". This is legal in NGSIv1.
• User does GET /v2/entities. What should CB return? CB shouldn't return "you&me" id, as & is a forbidden char in NGSIv2 API ids and returning that would break the API contract.

[kzangeli]

That is TRUE. Didn't think about that :-). Pity. So what to do? An entity with id you&me enters via APIv1, and we store it exactly like that in the database. Then a GET /v2/entities will receive the entity but with ID you_me. A little sad but OK I guess. Documentation necessary, of course. Perhaps we should even add an HTTP header to the response:

X-FIWARE-WARNING: Entity Ids modified

[fgalan]

strictNgsiv1Ids is deprecated functionality (see https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/deprecated.md) so, according to this rule

Bugs or issues related with deprecated features and not affecting any other feature are not addressed (they are closed in github.com as soon as they are spotted).

this issue is going to be closed.