perspective javascript API view.to_json() is not deterministic 😳

jordanskole commented 2 months ago

Bug Report

Steps to Reproduce:

create a table with data
render the table as a view, passing a config that will return a view with a schema that is different from the underlying table schema (expressions to create new table)
create an async function like this...

const SHARED_CONFIG = {...config}

const myAsyncOne = async () => {
  const view = await table.view(SHARED_CONFIG)
  const rows = await view.to_json();
  await view.delete()
  return rows;
}

const myAsyncTwo = async () => {
  const view = await table.view(SHARED_CONFIG)
  const rows = await view.to_json();
  const schema = await view.schema();
  await view.delete()
  return rows;
}

const isDeterministic = (one, two) => {
  return one == two;
} 

(async () => {

  const rowsOne = await myAsyncOne();
  const rowsTwo = await myAsyncTwo();
  const result = isDeterministic(); // expect true 

})()

Expected Result:

I would expect that the rows after being serialized to json would match, at least the properties should match, but that is not the case.

if you wait for the schema to come back each row will have a shape like:

{ data: { ...row } }

If you don't call the schema function you will end up with the "Raw" rows

{...row}

They are otherwise the same!

Actual Result:

brb!

...what actually happened, including full exceptions (please include the entire stack trace, including "caused by" entries), log entries, screen shots etc. where appropriate...

Environment:

I am currently on "@finos/perspective": "^2.8.0",

Additional Context:

...add any other context about the problem here. If applicable, add screenshots to help explain...

texodus commented 2 months ago

This is not how equivalence in JavaScript works

jordanskole commented 2 months ago

I know, I am not meaning strict equals (why I said "match"), i mean that somehow the row object transforms from schema to {data: schema}.

Another way to say the same thing would be that the Object.keys(row) are not even deterministic.

concrete examples:

obj1 = {
  firstName: "jordan", 
  lastName: "skole"
}

obj2 = {
  data: {
    firstName: "jordan", 
    lastName: "skole"
   }
 }

i know that even if the two had the same key/value pairs they would not strict equal each other, that's not what this is about

P.S. Im not sure it is a perspective bug, I am still investigating it still could be me!

P.P.S. I updated my original code example to remove the confusing triple equals

jordanskole commented 2 months ago

Here's a bit more detail:

These two lines are touching each other in the code:

const rows = await view.to_json();
console.log(rows);
const csv = await view.to_csv();
console.log(csv);

to_json:

[
    {
        "data": {
            "__ROW_PATH__": [],
            "smbId": 1136328,
            "firstPaymentDate": 1687627115560,
            "firstPaymentAmount": 2,
            "mostRecentPaymentDate": 1712836525823,
            "mostRecentPaymentAmount": 4,
            "totalPaymentCount": 364,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1136328
            ],
            "smbId": 1136328,
            "firstPaymentDate": 1687627115560,
            "firstPaymentAmount": 2,
            "mostRecentPaymentDate": 1687627115560,
            "mostRecentPaymentAmount": 2,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1147612
            ],
            "smbId": 1147612,
            "firstPaymentDate": 1695686375383,
            "firstPaymentAmount": 500,
            "mostRecentPaymentDate": 1695686375383,
            "mostRecentPaymentAmount": 500,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1148319
            ],
            "smbId": 1148319,
            "firstPaymentDate": 1691002664740,
            "firstPaymentAmount": 1477,
            "mostRecentPaymentDate": 1713640901023,
            "mostRecentPaymentAmount": 369.25,
            "totalPaymentCount": 31,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1149920
            ],
            "smbId": 1149920,
            "firstPaymentDate": 1691162946900,
            "firstPaymentAmount": 825,
            "mostRecentPaymentDate": 1704367713683,
            "mostRecentPaymentAmount": 2700,
            "totalPaymentCount": 8,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1153622
            ],
            "smbId": 1153622,
            "firstPaymentDate": 1692274214276,
            "firstPaymentAmount": 2000,
            "mostRecentPaymentDate": 1692274214276,
            "mostRecentPaymentAmount": 2000,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1154980
            ],
            "smbId": 1154980,
            "firstPaymentDate": 1693405161526,
            "firstPaymentAmount": 660,
            "mostRecentPaymentDate": 1713472581446,
            "mostRecentPaymentAmount": 575,
            "totalPaymentCount": 6,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1156225
            ],
            "smbId": 1156225,
            "firstPaymentDate": 1696548674700,
            "firstPaymentAmount": 505.05,
            "mostRecentPaymentDate": 1698189618586,
            "mostRecentPaymentAmount": 505.05,
            "totalPaymentCount": 2,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1156632
            ],
            "smbId": 1156632,
            "firstPaymentDate": 1694089245260,
            "firstPaymentAmount": 942,
            "mostRecentPaymentDate": 1713898586363,
            "mostRecentPaymentAmount": 2258.83,
            "totalPaymentCount": 14,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1157202
            ],
            "smbId": 1157202,
            "firstPaymentDate": 1693604761570,
            "firstPaymentAmount": 1500,
            "mostRecentPaymentDate": 1694129629550,
            "mostRecentPaymentAmount": 1500,
            "totalPaymentCount": 7,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1158667
            ],
            "smbId": 1158667,
            "firstPaymentDate": 1694114844040,
            "firstPaymentAmount": 20,
            "mostRecentPaymentDate": 1709305594560,
            "mostRecentPaymentAmount": 1932,
            "totalPaymentCount": 3,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1160871
            ],
            "smbId": 1160871,
            "firstPaymentDate": 1700570537133,
            "firstPaymentAmount": 400,
            "mostRecentPaymentDate": 1707406315910,
            "mostRecentPaymentAmount": 50,
            "totalPaymentCount": 4,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1166428
            ],
            "smbId": 1166428,
            "firstPaymentDate": 1697656388703,
            "firstPaymentAmount": 633,
            "mostRecentPaymentDate": 1713202539476,
            "mostRecentPaymentAmount": 500,
            "totalPaymentCount": 71,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1168107
            ],
            "smbId": 1168107,
            "firstPaymentDate": 1698069185866,
            "firstPaymentAmount": 60,
            "mostRecentPaymentDate": 1708437910776,
            "mostRecentPaymentAmount": 100,
            "totalPaymentCount": 85,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1170172
            ],
            "smbId": 1170172,
            "firstPaymentDate": 1698442577433,
            "firstPaymentAmount": 2000,
            "mostRecentPaymentDate": 1712880228230,
            "mostRecentPaymentAmount": 2000,
            "totalPaymentCount": 7,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1186560
            ],
            "smbId": 1186560,
            "firstPaymentDate": 1703089411616,
            "firstPaymentAmount": 2000,
            "mostRecentPaymentDate": 1703797646130,
            "mostRecentPaymentAmount": 1300,
            "totalPaymentCount": 2,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1192518
            ],
            "smbId": 1192518,
            "firstPaymentDate": 1704229847566,
            "firstPaymentAmount": 65,
            "mostRecentPaymentDate": 1709061801590,
            "mostRecentPaymentAmount": 100,
            "totalPaymentCount": 5,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1192923
            ],
            "smbId": 1192923,
            "firstPaymentDate": 1709828955806,
            "firstPaymentAmount": 32,
            "mostRecentPaymentDate": 1713443871336,
            "mostRecentPaymentAmount": 15.54,
            "totalPaymentCount": 5,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1196669
            ],
            "smbId": 1196669,
            "firstPaymentDate": 1705407905636,
            "firstPaymentAmount": 235,
            "mostRecentPaymentDate": 1712946508483,
            "mostRecentPaymentAmount": 1150,
            "totalPaymentCount": 6,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1197533
            ],
            "smbId": 1197533,
            "firstPaymentDate": 1706651586056,
            "firstPaymentAmount": 45,
            "mostRecentPaymentDate": 1713903750766,
            "mostRecentPaymentAmount": 55,
            "totalPaymentCount": 47,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1197554
            ],
            "smbId": 1197554,
            "firstPaymentDate": 1707307534540,
            "firstPaymentAmount": 300,
            "mostRecentPaymentDate": 1713494181453,
            "mostRecentPaymentAmount": 37,
            "totalPaymentCount": 50,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1212184
            ],
            "smbId": 1212184,
            "firstPaymentDate": 1709835584773,
            "firstPaymentAmount": 933.75,
            "mostRecentPaymentDate": 1709835584773,
            "mostRecentPaymentAmount": 933.75,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1212230
            ],
            "smbId": 1212230,
            "firstPaymentDate": 1707863234223,
            "firstPaymentAmount": 1500,
            "mostRecentPaymentDate": 1707863234223,
            "mostRecentPaymentAmount": 1500,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1221757
            ],
            "smbId": 1221757,
            "firstPaymentDate": 1708983206610,
            "firstPaymentAmount": 525,
            "mostRecentPaymentDate": 1708983206610,
            "mostRecentPaymentAmount": 525,
            "totalPaymentCount": 1,
            "paymentStatus": "Active    "
        }
    },
    {
        "data": {
            "__ROW_PATH__": [
                1237009
            ],
            "smbId": 1237009,
            "firstPaymentDate": 1711472796663,
            "firstPaymentAmount": 4,
            "mostRecentPaymentDate": 1712836525823,
            "mostRecentPaymentAmount": 4,
            "totalPaymentCount": 5,
            "paymentStatus": "Active    "
        }
    }
]

to_csv:

"smbId (Group by 1)","smbId","firstPaymentDate","firstPaymentAmount","mostRecentPaymentDate","mostRecentPaymentAmount","totalPaymentCount","paymentStatus"
,1136328,2023-06-24 17:18:35.560,2,2024-04-11 11:55:25.823,4,364,"Active    "
1136328,1136328,2023-06-24 17:18:35.560,2,2023-06-24 17:18:35.560,2,1,"Active    "
1147612,1147612,2023-09-25 23:59:35.383,500,2023-09-25 23:59:35.383,500,1,"Active    "
1148319,1148319,2023-08-02 18:57:44.740,1477,2024-04-20 19:21:41.023,369.25,31,"Active    "
1149920,1149920,2023-08-04 15:29:06.900,825,2024-01-04 11:28:33.683,2700,8,"Active    "
1153622,1153622,2023-08-17 12:10:14.276,2000,2023-08-17 12:10:14.276,2000,1,"Active    "
1154980,1154980,2023-08-30 14:19:21.526,660,2024-04-18 20:36:21.446,575,6,"Active    "
1156225,1156225,2023-10-05 23:31:14.700,505.05,2023-10-24 23:20:18.586,505.05,2,"Active    "
1156632,1156632,2023-09-07 12:20:45.260,942,2024-04-23 18:56:26.363,2258.83,14,"Active    "
1157202,1157202,2023-09-01 21:46:01.570,1500,2023-09-07 23:33:49.550,1500,7,"Active    "
1158667,1158667,2023-09-07 19:27:24.040,20,2024-03-01 15:06:34.560,1932,3,"Active    "
1160871,1160871,2023-11-21 12:42:17.133,400,2024-02-08 15:31:55.910,50,4,"Active    "
1166428,1166428,2023-10-18 19:13:08.703,633,2024-04-15 17:35:39.476,500,71,"Active    "
1168107,1168107,2023-10-23 13:53:05.866,60,2024-02-20 14:05:10.776,100,85,"Active    "
1170172,1170172,2023-10-27 21:36:17.433,2000,2024-04-12 00:03:48.230,2000,7,"Active    "
1186560,1186560,2023-12-20 16:23:31.616,2000,2023-12-28 21:07:26.130,1300,2,"Active    "
1192518,1192518,2024-01-02 21:10:47.566,65,2024-02-27 19:23:21.590,100,5,"Active    "
1192923,1192923,2024-03-07 16:29:15.806,32,2024-04-18 12:37:51.336,15.54,5,"Active    "
1196669,1196669,2024-01-16 12:25:05.636,235,2024-04-12 18:28:28.483,1150,6,"Active    "
1197533,1197533,2024-01-30 21:53:06.056,45,2024-04-23 20:22:30.766,55,47,"Active    "
1197554,1197554,2024-02-07 12:05:34.540,300,2024-04-19 02:36:21.453,37,50,"Active    "
1212184,1212184,2024-03-07 18:19:44.773,933.75,2024-03-07 18:19:44.773,933.75,1,"Active    "
1212230,1212230,2024-02-13 22:27:14.223,1500,2024-02-13 22:27:14.223,1500,1,"Active    "
1221757,1221757,2024-02-26 21:33:26.610,525,2024-02-26 21:33:26.610,525,1,"Active    "
1237009,1237009,2024-03-26 17:06:36.663,4,2024-04-11 11:55:25.823,4,5,"Active    "

texodus commented 2 months ago

@jordanskole We can only help you with a full reproduction that I can replicate, a description of the output you observed is insufficient.

to_json() has hundreds of tests, if you need a reference for what a repro looks like.

jordanskole commented 2 months ago

Understood.

finos / perspective