search

ArctosDB / dev

Current public release and project roadmap for development
https://arctos.database.museum
Apache License 2.0
0 stars 0 forks source link

Request - un-type component loaders #110

Closed dustymc closed 1 week ago

dustymc commented 4 weeks ago

Help us understand your request (check below):

Describe what you're trying to do

In trying to find data for https://github.com/ArctosDB/dev/issues/94, I downloaded (from prod) and attempted to upload (in test), and....

ERROR: invalid input syntax for type double precision: ""

Even ignoring external tools which have their own crazy, there are a bunch of tools at various levels in the component loader ecosystem and they don't always play nice with each other. In this case a CF library is sending "nothing, but in a perfectly valid quote-everything dialect of CSV" and a postgres client can't interpret that as the intended ~NULL::numeric.

I think this may be a semi-common source of user frustration, although I haven't been able to fully recreate until now. I suspect converting all fields in all component loaders to text and adding in explicit type checks when necessary would be a usability upgrade, so I propose to establish this as an Arctos Standard and begin rebuilding tools.

Possibly that's short-sighted and this would just be a way to cause sneaky problems/errors/corruptions, datatypes exist for very legitimate reasons after all.

Help?

dustymc commented 2 weeks ago

Enthusiastically supportive silence accepted, adopting this beginning with https://github.com/ArctosDB/dev/issues/105.