catmaid / CATMAID

Collaborative Annotation Toolkit for Massive Amounts of Image Data
https://catmaid.org
GNU General Public License v3.0
185 stars 56 forks source link

Support 64 Bit IDs in front-end #1963

Open tomka opened 4 years ago

tomka commented 4 years ago

With almost all back-end IDs now being 64 Bit integers, the front-end needs to catch up at some point and use BigInt64 where appropriate. Some discussion on that happened already in PR #1887. A central target for such a change would be the JSON parsing in static/libs/catmaid/CATMAID.js for CATMAID.fetch(). There are likely more places that require an update though.

clbarnes commented 4 years ago

So how does this work when we just get JSON back? Not all numbers in the response will want to be ints, obviously. The frontend needs to know the type of the number, whether that's through explicit casts at parse time given an expected response, or the response itself containing type information. JSON obviously can't distinguish ints and floats: msgpack can but probably doesn't cast to bigints by default. Typescript does have a bigint type but that information isn't used at runtime.

aschampion commented 4 years ago

Unfortunately the 64-bit ints may need to go back to being strings in the JSON, which then get parsed into BigInt64s at the specific sites where they're used. You can manually replace JSON.parse with something that uses another type (in that case BigNumber), but (a) that replaces an optimized, browser-provided function with a user code one, and (b) we don't want to replace all numeric types, but adapt based on the schema of the JSON.

However, we hopefully have some lead time on this until we start approaching the 53 bit barrier in production instances, so maybe we'll find custom parsing viable before then.

aschampion commented 4 years ago

Perhaps a good first step in this would instead be considering the class of bugs that could be introduced by the front-end corrupting IDs, and making sure we've preventing the most catastrophic outcomes. Generally we place a lot of trust in the front-end, and while we're all in agreement that we don't care about defending from malicious front-ends or users, we could possibly do more to address faulty ones, similar in spirit to state checking.

aschampion commented 4 years ago

Pinging #1955 because a lot of segmentation data uses numbers neighboring 2^64 as special semantic labels, so if we want to directly support those in label stacks that may be our first instance of backend<>frontend communication needing a working 64 bit solution.