orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
37 stars 49 forks source link

Remove the null terminator from flask_utils.py #709

Open NEC-Vishal opened 1 year ago

NEC-Vishal commented 1 year ago

Proposed changes

Types of changes

What types of changes does your code introduce to the project? Put an x in the boxes that apply

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

"Martel Open Source Software Individual Contributor License Agreement"
"Contributing to QuantumLeap"
"QuantumLeap Release Notes"
github-actions[bot] commented 1 year ago

CLA Assistant Lite bot All contributors have signed the CLA ✍️

NEC-Vishal commented 1 year ago

707

c0c0n3 commented 1 year ago

@NEC-Vishal before going ahead w/ this PR, can you please have a read through #707. I realised the TODO in the code isn't really helpful, it's so vague that's basically impossible to understand. #707 now explains the whole thing in detail.

c0c0n3 commented 1 year ago

@NEC-Vishal another thing I realised is that if we merged the small change you implemented, the whole WQ query API would break. This is my fault b/c I didn't realise we needed to have test cases to cover the array streaming format. So let's add test cases. Can please create a Python package wq.ql.tests and a file test_flaskutils.py with the following content

import json
import pytest

from wq.ql.flaskutils import *

class Item(BaseModel):
    id: int

    @staticmethod
    def range(stop: int) -> ['Item']:
        return [Item(id=x) for x in range(stop)]

items_array_supply = [
    Item.range(stop) for stop in range(4)
    # [], [Item(id=1)], [Item(1), Item(2)], ...
]

def stream_to_json_doc(xs: Iterable[BaseModel]) -> str:
    lines = [line for line in json_array_streamer(xs)]
    return "".join(lines)

def extract_item_ids(streamed_doc: str) -> [int]:
    item_array = json.loads(streamed_doc)
    return [x['id'] for x in item_array if x is not None]

@pytest.mark.parametrize('items', items_array_supply)
def test_json_array_streamer(items: [Item]):
    want = [x.id for x in items]
    got = extract_item_ids(stream_to_json_doc(items))

    assert want == got
c0c0n3 commented 1 year ago

@NEC-Vishal then you should hook up the new tests to the ql suite. If you run the tests in that file against your code that zaps the JSON null value, you should see a JSON decode exception.