[API/BUG] requests latest alerts by class with some v: ... restricted columns bug

[FusRoman]

I wanted to request fink to get the latests alerts from fink and return only a subset of columns. In the subset of columns, there were columns added by fink (v: ...) but the request failed with those columns and not failed when i removed the v: ... columns.

Buggy code

r = requests.post(
    "https://fink-portal.org/api/v1/latests",
    json={
        "class": "SN candidate",
        "n": "5",
        "columns": "i:ra, i:dec, v:classification"
    },
)

print(pd.read_json(r.content))

Output

Traceback (most recent call last):
  File "processor.py", line 54, in <module>
    print(pd.read_json(r.content))
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 207, in wrapper
    return func(*args, **kwargs)
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 614, in read_json
    return json_reader.read()
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 748, in read
    obj = self._get_object_parser(self.data)
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 770, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 885, in parse
    self._parse_no_numpy()
  File "/home/roman.le-montagner/.local/lib/python3.7/site-packages/pandas/io/json/_json.py", line 1140, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value

Not buggy

r = requests.post(
    "https://fink-portal.org/api/v1/latests",
    json={
        "class": "SN candidate",
        "n": "5"
    },
)

print(pd.read_json(r.content))

Output

  d:cdsxmatch  d:mulens  d:rf_kn_vs_nonkn  d:rf_snia_vs_nonia  d:roid  ...     i:ypos  d:tracklet  v:classification               v:lastdate  v:constellation
0     Unknown         0                 0               0.000       0  ...   172.9430                  SN candidate  2022-04-20 12:11:56.999          Pegasus
1     Unknown         0                 0               0.046       0  ...  1642.8579                  SN candidate  2022-04-20 12:11:56.999        Andromeda
2     Unknown         0                 0               0.046       0  ...  1641.1895                  SN candidate  2022-04-20 12:05:06.996        Andromeda
3     Unknown         0                 0               0.046       0  ...   784.0905                  SN candidate  2022-04-20 12:02:23.001        Andromeda
4     Unknown         0                 0               0.000       0  ...   169.8038                  SN candidate  2022-04-20 12:01:42.004          Pegasus

[5 rows x 47 columns]

We can see that the v: ... values are returned when we don't restrict the columns but if we want only a subset of columns with some v: ... columns, the code exit with error.

[JulienPeloton]

Thanks for the report. Indeed the error arises because v:classification is a field computed on the fly in the server, and it requires several columns to be present:

https://github.com/astrolabsoftware/fink-science-portal/blob/472a687adb8470a5c8f53da4a3991e538017dafe/apps/utils.py#L71-L87

We could resolve the problem by transferring the required columns under the hood to compute v:classification, and remove them in the final output. I will propose a fix.

[FusRoman]

Thanks @JulienPeloton :+1:

[anaismoller]

I add a + to "important bug" :) thanks!