search

koopjs / FeatureServer

An open source Geoservices Implementation (deprecated)
https://geoservices.github.io
Other
104 stars 32 forks source link

Support 64 bit ObjectIds #121

Closed jkerr5 closed 5 years ago

jkerr5 commented 5 years ago

According to https://developers.arcgis.com/rest/services-reference/layer-feature-service-.htm

If the layer objectIdField does not have a length property or the length property is set to 4, the objectIdField is 32-bit. If the objectIdField has a length of 8, the objectIdField is 64-bit.

Can you add support for specifying the length of the ObjectId? Maybe by allowing the idFieldLength in the metadata?

rgwozdz commented 5 years ago

@jkerr5 thanks for posting this and providing the above link. I will put together a PR for 64 bit support in both FeatureServer and Winnow.

rgwozdz commented 5 years ago

Perhaps we could have the OBJECTID always be 64 bit?

jkerr5 commented 5 years ago

That would be fine with me.

jkerr5 commented 5 years ago

@rgwozdz any thoughts on when this might make it into a release?

rgwozdz commented 5 years ago

Hopefully relatively soon. I'm juggling it with a few other things. It's going to require a companion update/release of Winnow and then testing on all the ArcGIS clients, so it's a bit of work to coordinate - unless someone else wants the PR, I probably can't get to it until end of the week.

jkerr5 commented 5 years ago

Thanks for the update @rgwozdz

rgwozdz commented 5 years ago

@jkerr5 - I'm realizing there is an impediment to implementation. JavaScript represents numbers using IEEE-754 double-precision (64 bit) format. It turns out that that give sus only 53 bits precision, which is around fifteen to sixteen decimal digits. When you get to numbers that large or larger, you end up getting approximations:

var bigNumber = 9023372036854775807
console.log(bigNumber); // 9023372036854776000

So while the link you provide indicates ArcGIS support of 64-bit OBJECTIDs, Koop can't readily deliver them due to limitations of JavaScript. A provider could supply a 64-bit integer as a string, but it would have to be converted to an integer prior to serving it to ArcGIS clients (as OBJECTIDs appear to still have to be integers for ArcGIS clients). That conversion could lead to a loss of precision, with the potential result being duplicate OBJECTIDs.

One way forward would be to have FeatureServer set the OBJECTID length to 8 (indicating 64-bit) but print a warning if a provider pushed through an OBJECTID that was greater that the unsigned 53-bit max value (9007199254740991).

How many digits where you hoping to support? Wondering if a max OBJECTID value of 9007199254740991 will suit your needs prior to undertaking the work.

jkerr5 commented 5 years ago

The 53-bit max value of 9007199254740991 would be good. Anything more than the 32-bits we have now would be great :)

On Fri, Oct 12, 2018 at 7:47 PM Rich Gwozdz notifications@github.com wrote:

@jkerr5 https://github.com/jkerr5 - I'm realizing there is an impediment to implementation. JavaScript represents numbers using IEEE-754 double-precision (64 bit) format. It turns out that that give sus only 53 bits precision, which is around fifteen to sixteen decimal digits. When you get to numbers that large or larger, you end up getting approximations:

var bigNumber = 9023372036854775807 console.log(bigNumber); // 9023372036854776000

So while the link you provide indicates ArcGIS support of 64-bit OBJECTIDs, Koop can't readily deliver them due to limitations of JavaScript. A provider could supply a 64-bit integer as a string, but it would have to be converted to an integer prior to serving it to ArcGIS clients (as OBJECTIDs appear to still have to be integers for ArcGIS clients). That conversion could lead to a loss of precision, with the potential result being duplicate OBJECTIDs.

One way forward would be to have FeatureServer set the OBJECTID length to 8 (indicating 64-bit) but print a warning if a provider pushed through an OBJECTID that was greater that the unsigned 53-bit max value (9007199254740991).

How many digits where you hoping to support? Wondering if a max OBJECTID value of 9007199254740991 will suit your needs prior to undertaking the work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/koopjs/FeatureServer/issues/121#issuecomment-429406195, or mute the thread https://github.com/notifications/unsubscribe-auth/AAP8Ais_yHOr_9otDds9PLHVkz8dCYPhks5ukNXLgaJpZM4Wu-S8 .

jkerr5 commented 5 years ago

Hi @rgwozdz Any update on an ETA for this? We are working a project that needs bigger OBJECTIDs so were hoping this would make it in soon. Thanks.

rgwozdz commented 5 years ago

Hey @jkerr5 - sorry, I've been consumed with a few other big projects so I haven't been able to get to it. I may be freed up starting next week. Sorry, if you guys want to start your own PR, feel free. But it's still in my queue.

rgwozdz commented 5 years ago

@jkerr5 - finally got around to prototyping this, but I am finding that it doesn't work despite the documentation you noted in https://developers.arcgis.com/rest/services-reference/layer-feature-service-.htm.

I find that even when the length of the idField is set to 8, values over the 2147483647 max 32bit limit cause buggy behavior in ArcGIS Pro. Specifically, I'm finding that in the attribute table, the idField is assigned a value of 0 even though koop is sending a value of say 2147483650. Really large numbers appear to cause Pro to crash.

jkerr5 commented 5 years ago

That's unfortunate to hear. I posted a question to the community https://community.esri.com/message/821822-can-arcgis-pro-support-64-bit-objectids

rgwozdz commented 5 years ago

Ideally, we would have an example feature service that implements the objectIdField length property and pumps out 64 bit integer IDs. Then we could analyze the responses, test in clients, and simulate the behavior in Koop.

rgwozdz commented 5 years ago

@jkerr5 - closing this given the response at https://community.esri.com/message/821822-can-arcgis-pro-support-64-bit-objectids