search

yapstudios / YapDatabase

YapDB is a collection/key/value store with a plugin architecture. It's built atop sqlite, for Swift & objective-c developers.
Other
3.35k stars 365 forks source link

Best way for importing data from "outside"? #372

Closed mickeyl closed 7 years ago

mickeyl commented 8 years ago

I'm very satisfied with YapDatabase on the client. Alas, for some of my projects, I have the necessity to start with a large prepopulated database. It seems pretty cumbersome to write a dedicated data importer in Objective-C. I'd rather use a scripting language, e.g. Python, to generate my database upfront, possibly on a beefy machine that's not running macOS. Are bindings to other programming languages planned or feasible?

chrisballinger commented 8 years ago

It might be possible to run via GNUstep on Linux and then write bindings for it, but it would probably be easiest to write a macOS tool that shares code with your iOS app.. or just do it in your iOS app and extract the prepopulated db (that's what we do in iBurn).

For a pure Python tool you'd also need to rewrite your object serialization/deserialization code e.g. NSCoding/NSKeyedArchiver, and then also rewrite all the code to do all of the Yap specific stuff like view registration etc. I think Robbie was working on a C++ port at one point but I'm not sure if it was meant to be 100% compatible with the db format, or just using similar ideas.

cipherCOM commented 8 years ago

Hi @mickeyl!

We did something similar like @chrisballinger proposed. We had a special "create seeddatabase" target for our app. This had special instructions on how to fill the local DB. Afterwards we would extract it from the simulator and vacuum it. Ours was around 50-100MB and was mainly limited in performance by web service calls to acquire the data. If you dislike ObjC you could also use Swift in this target ;)

Of course there are tools like PyObjC but I personally would advise against a complicated and brittle solution like this.

mickeyl commented 8 years ago

Thanks for the pointers. Yes, I guess writing special ObjC code for populating the database seems the easiest approach. I will probably try to offload the most complicated stuff into a dedicated (Python) script that puts out a JSON file that contains everything parsed into model objects right away which then only needs to be parsed by ObjC and stuck into the database.