API for linking MotionModels to Data Stores

[sxross]

New thread to discuss API for linking MotionModels to data stores, e.g., ParseModel and input presenters, e.g. Formotion. See issue #12.

[DougPuchalski]

I'm beginning to look at this today for my project.

I've been using NanoStore and NanoStoreInMotion and it works pretty well, but want to leverage MotionModel since I like the Rails API and the fact that it is storage agnostic.

[adelevie]

+1

It would be great to be able to use Parse for storage.

Sent from my iPhone

On Feb 14, 2013, at 1:48 PM, Doug Puchalski notifications@github.com wrote:

I'm beginning to look at this today for my project.

I've been using NanoStore and NanoStoreInMotion and it works pretty well, but want to leverage MotionModel since I like the Rails API and the fact that it should be more storage agnostic.

— Reply to this email directly or view it on GitHub.

[DougPuchalski]

I'm focusing on sqlite3, but I think a base adapter model that can be subclassed is the way to go.

[adelevie]

Storage will probably be pretty straightforward, but querying could be tough.

Sent from my iPhone

On Feb 14, 2013, at 1:54 PM, Doug Puchalski notifications@github.com wrote:

I'm focusing on sqlite3, but I think a base adapter model that can be subclassed is the way to go.

— Reply to this email directly or view it on GitHub.

[sxross]

Factoring out the storage aspect of MotionModel is very much in keeping with its future direction. Right now, MotionModel relies on NSCoder to serialize data. It then uses NSKeyedArchiver to create an NSData representation. This is a default implementation. The idea was to make it simple to manage storage-light information. It's grown since then :)

[sxross]

Note to Alan: The FinderQuery class hits each relational operator as a method. In its current implementation, MotionModel actually filters down the result set. However, it could just as easily build a query string to be passed to something like SQLite.

[adelevie]

Also re: parse, their sdk does all of the heavy lifting...getting it to play nice with motion model is just a matter of wiring.

Sent from my iPhone

On Feb 14, 2013, at 1:58 PM, "s.ross" notifications@github.com wrote:

Factoring out the storage aspect of MotionModel is very much in keeping with its future direction. Right now, MotionModel relies on NSCoder to serialize data. It then uses NSKeyedArchiver to create an NSData representation. This is a default implementation. The idea was to make it simple to manage storage-light information. It's grown since then :)

— Reply to this email directly or view it on GitHub.

[DougPuchalski]

For the inspired, please see https://github.com/aceofspades/MotionModel/tree/sqlite

Persistence is split into adapters, and basic functionality is working for SQLite3 using FMDB. No specs as of yet. I'll be moving on to associations next.

Note that models now need to mix in the appropriate model adapter for this to work.

[DougPuchalski]

Those who are interested in other adapters, i.e. Parse, please see https://github.com/sxross/MotionModel/pull/22, which is a first step to get to https://github.com/aceofspades/MotionModel/tree/sqlite. The sqlite branch however might be a better place to start, as it attempts to set up a framework for different adapters and a model proxy.

[sxross]

This is really exciting stuff. I believe the NSDate stuff needs fixing. I've been postponing it hoping some kind soul would port Chronic or something like it over to Motion. The current thinking behind date/time handling is that when casting an arbitrary string to a date/time, we should get as close to arbitrary as possible. Maybe that's not realistic, but that was my goal. There is no dependency on BubbleWrap that I'm aware of, however.

I had begun work on refactoring MotionModel::Model to pull volatile storage (i.e., collection) out of the model object itself and put it into its own service module. I was considering a REST-style interface, but it looks like you've chosen a protocol-style one. That might be more efficient -- I'd like to hear your thoughts.

When you're ready to consider moving this into Motion, please let me know. I don't know, but I suspect somebody out there might be relying on the NSCoder persistence adaptor to remain file-compatible with existing MotionModel data stores, so it would be nice to reassure ourselves that this is true.

Thanks for working on this. As I said, very exciting.

[DougPuchalski]

Indeed, I'm pretty excited to get this far so quickly. Thanks for the solid starting point!

Agreed NSDate is not fully baked. If you try to compile my test project without including bubble-wrap, you'll see exceptions when you build. Basically NSDate.dateWithNaturalLanguageString is not part of iOS, I have no idea how that's even resolving, but it does with BW.

My personal opinion as far as dates is that integer timestamps should be stored. There's no ambiguity or formatting confusion. Leave it up to the app to decide on presentation or valid input formats (i18n). With Sqlite3 anyway, there's no native date format, so it's either numeric or text. Text is nice when looking at the raw DB but it can also be ambiguously formatted (which format is that again? let me look that up...etc.) Maybe you could parse strings into integers coming in, but I would say only in one format, i.e. ISO 9075 like MySQL. Again otherwise you have to guess and there will be bugs!

Not entirely sure how a REST interface would look, I usually go there thinking about networks and services. Making an adapter seemed like the easiest approach to preserve your code and open the door for other adapters. Rails uses adapters. Once there's an adapter, you can build whatever you want. I don't think my approach adds very much efficiency hit beyond some method delegation overhead. Probably insignificant given SQL or a network adapter, maybe a little hit for your Array collection is somewhat noticeable.

Basically I am motivated by Rails because it has a battled hardened API, and people already know it. No need to reinvent the wheel or add confusion going back and forth between mobile and web. After dealing with many iOS libraries, I'd prefer to use as much ruby as possible and only use NS where necessary or for major efficiency gain. I think FMDB can do a lot more but again, more ruby is easier to understand, change, and enhance. You'll notice my query syntax is more like ActiveRecord that your FinderQuery is for this reason. Basically I'd like to try to support the Rails API unless there's a compelling reason not to.

As soon as you've had a chance to digest this, I think the first step might be to merge #22 if you're comfortable that it hasn't broken anything. At that point, maybe we can pull sqlite into a new branch and get some contributions in there. No rush pulling to master, let's just keep it compatible. The sql branch will certainly need tests but honestly I'm not sure how much time I'll be able to devote to that given my schedule. Maybe somebody can help with that.

[DougPuchalski]

To be sure it's not missed, I think the major version number needs to be bounced for #22, as it will break existing models that will need to mix in the ArrayAdapter module.

[sxross]

NSDate:

I believe Apple relies way too much on NSDateFormatter and it makes dealing with dates/times and durations a PITA. I'd love to see Ruby implementations of good date/time handling. Although NSDate.dateWithNaturalLanguageString is not officially part of iOS, and even though there is suspicion Apple may get irritated if we still use it, I'm reluctant to remove it until there's a suitable replacement. I'm really missing a lot of Ruby's Time and DateTime stuff. What's up with that @lrz?

Date/time storage:

I've wrestled with this again and again (and again last night :). Rails works with the premise that absolutely every input will be text because that's how it arrives over HTTP. For storage, I believe Rails delegates this to the adaptor and is agnostic about it. The current NSCoding implementation uses NSDate, primarily because when you have one of these, it can make things like date pickers easier to bring up. There may be a bit of naivety of all things iOS showing through here on may part.

REST interface:

I believe this would be inherently less efficient on the client, but more transparent to any remote. At least, that was my scratchpad thinking when I came up with it. But REST is a protocol defined by a service provider, so I expect what you've defined is at least as efficient if not more so.

Rails API

I agree that Rails is a good model (at least the part of Rails that applies here). ActiveRecord as it was originally implemented was heavily SQL flavored, which makes it inappropriate for certain storage scenarios. For example: Person.find(:first, :conditions => "age < 27 and name like %jon%'") would have to be parsed in MotionModel. However, DataMapper and ActiveModel take a slightly different approach, implementing relational methods that you chain. That's the pattern I followed in MotionModel because it seemed particularly Rubyish and very implementation-agnostic. I don't believe I've strayed so far from Rails that people will be completely at sea with the syntactic sugar in MotionModel. If you see areas for improvement, please let me know. (I did, in fact, copy and paste a model from MM to a Rails app and with a small amount of editing had an operational version.)

Thanks again for this and I'll have a close look at #22 (I've only had a short look and what I see is very exciting).

On Feb 17, 2013, at 12:00 PM, Doug Puchalski notifications@github.com wrote:

Indeed, I'm pretty excited to get this far so quickly. Thanks for the solid starting point!

Agreed NSDate is not fully baked. If you try to compile my test project without including bubble-wrap, you'll see exceptions when you build. Basically NSDate.dateWithNaturalLanguageString is not part of iOS, I have no idea how that's even resolving, but it does with BW.

My personal opinion as far as dates is that integer timestamps should be stored. There's no ambiguity or formatting confusion. Leave it up to the app to decide on presentation or valid input formats (i18n). With Sqlite3 anyway, there's no native date format, so it's either numeric or text. Text is nice when looking at the raw DB but it can also be ambiguously formatted (which format is that again? let me look that up...etc.) Maybe you could parse strings into integers coming in, but I would say only in one format, i.e. ISO 9075 like MySQL. Again otherwise you have to guess and there will be bugs!

Not entirely sure how a REST interface would look, I usually go there thinking about networks and services. Making an adapter seemed like the easiest approach to preserve your code and open the door for other adapters. Rails uses adapters. Once there's an adapter, you can build whatever you want. I don't think my approach adds very much efficiency hit beyond some method delegation overhead. Probably insignificant given SQL or a network adapter, maybe a little hit for your Array collection is somewhat noticeable.

Basically I am motivated by Rails because it has a battled hardened API, and people already know it. No need to reinvent the wheel or add confusion going back and forth between mobile and web. After dealing with many iOS libraries, I'd prefer to use as much ruby as possible and only use NS where necessary or for major efficiency gain. I think FMDB can do a lot more but again, more ruby is easier to understand, change, and enhance. You'll notice my query syntax is more like ActiveRecord that your FinderQuery is for this reason. Basically I'd like to try to support the Rails API unless there's a compelling reason not to.

As soon as you've had a chance to digest this, I think the first step might be to merge #22 if you're comfortable that it hasn't broken anything. At that point, maybe we can pull sqlite into a new branch and get some contributions in there. No rush pulling to master, let's just keep it compatible. The sql branch will certainly need tests but honestly I'm not sure how much time I'll be able to devote to that given my schedule. Maybe somebody can help with that.

— Reply to this email directly or view it on GitHub.

[DougPuchalski]

Where does NSDate.dateWithNaturalLanguageString come from? Good point, I heard Apple scans for undocumented API calls and rejects apps for it.

As long as the app can control which formats for incoming date parsing, and as long as the default is one and only one format (i.e. ISO) I think that's reasonable.

Per Rails syntax, there were some differences I noticed in your syntax, and certainly the Array calls for differences over SQL -- which is why I made the SQLScope class. Preserve your syntax so it can be optimized (or converged if you see fit) but get as close to ActiveRecord as possible for SQL.

Oh yeah, by the way, I did consider whether this should be a separate library called MotionRecord or something. But I think this implementation will be much trimmer than Rails and probably fits in nicely here if you agree.

[sxross]

NSDate.dateWithNaturalLanguageString is a MacOS API, and it is implemented on iOS. I believe at some point Apple thought it was a bad idea and removed it from the documentation, but there is no deprecation or notation of such. It's commonly referred to on the 'net, as is NSDateFormatter. I'd prefer to get away from NS-anything except NSCoder for the coding adaptor. The problem I'm considering is this:

UIDatePicker#date is an NSDate. UILabel#text is a String (or probably an NSString)

If people declare a model with:

columns :birth_date => :date

They would (correctly) expect to present this in Formotion as a datepicker, but note that Formotion also presents a text entry box. I'm not wedded to Formotion, but there is an upside to allowing users to enter random date information as an alternative to using the picker just like there is an upside to typing cd ~/Desktop && ls -l as opposed to "go to the finder, switch to your Desktop directory, look at the file listing, switch to details mode".

So my thinking was, yeah, Formotion gets this right. But I need to somehow stuff that random text into some kind of a Date storage form that will allow for both compactness and future manipulation (such as comparison, difference, etc.). That's where NSDate.dateWithNaturalLanguageString crept in. As I said, this is an evolving project, and it illustrates a bit of my own iOS naivety, having a lot of Rails but very little iOS background.

I'm totally open to creating a "breaking change" that insists on ISO formatted dates as long as we provide some capacity for supporting current behavior, but I'm still not clear what the best format for (esp.) date/time is. That's why I was hoping for Chronic...

I'm planning to devote some time to building my own app right now but I promise to look over your work very soon. Your input has been incredible and I want to include it as soon as possible in MotionModel.

On Feb 17, 2013, at 1:22 PM, Doug Puchalski notifications@github.com wrote:

Where does NSDate.dateWithNaturalLanguageString come from? I think iOS has NSDateFormatter natively. Good point, I heard Apple scans for undocumented API calls and rejects apps for it.

As long as the app can control which formats for incoming date parsing, and as long as the default is one and only one format (i.e. ISO) I think that's reasonable.

Per Rails syntax, there were some differences I noticed in your syntax, and certainly the Array calls for differences over SQL -- which is why I made the SQLScope class. Preserve your syntax so it can be optimized (or converged if you see fit) but get as close to ActiveRecord as possible for SQL.

Oh yeah, by the way, I did consider whether this should be a separate library called MotionRecord or something. But I think this implementation will be much trimmer than Rails and probably fits in nicely here if you agree.

— Reply to this email directly or view it on GitHub.

[DougPuchalski]

I'm still not so sure. Single Responsibility would be that you wouldn't something like bundle complex date handling in with something as specific as a persistence implementation. Mapping to and from compatible formats is one thing, but doing interpretations, natural language, or i18n is pretty far out of scope IMO. But that doesn't mean I don't think you should want it! It should go into sugarcube or something and then an adapter (again!) here could mix it in on demand. It's more useful in a general sense all over an app, not just here specifically. And you want to share the same code everywhere.

I remain unsure of how BubbleWrap makes your methods available, again without it the sample won't find it.

Thanks for the conversation.

[DougPuchalski]

FYI significant refactoring is coming for https://github.com/aceofspades/MotionModel/tree/sqlite. It should be considered a demonstration/quick-start for now.

[DougPuchalski]

I've updated https://github.com/aceofspades/MotionModel/tree/sqlite and there are a few specs as well. There are a few unrelated spec failures that I'll get to when possible and then make a pull request.

With this version, I have scoping, order by, limit, associations, virtual attributes, and polymorhism working.

[sxross]

API now exists in the form of Adapters.