Future of clorm/Usecases/Views

[MaxOstrowski]

I like using clorm a lot, and it simplifies a lot of steps that I need to do on a daily basis, thanks.

I recently found a new viewing angle to clorm which for me basically splits this project in two parts. One is the awesome query language that operates on a database of python objects. The other functionality is to convert such a database to clingo terms (and the other way around, convert a model or instance - into python objects). Do you think this view makes sense? Do you think it makes sense to use already existing ORM systems for the first part and only provide a mapping to clingo objects if needed ? S.t. clorm + ORM system X has the same functionality as before, but the maintenance of the ORM and query stuff is with the external ORM system.

[daveraja]

@MaxOstrowski I think it does make sense to view the factbase/query stuff separately to conversion to clingo. There have been a number of cases where I've used clorm in a non-ASP related setting because I wanted to use the query mechanism, rather than create some messy data-structure.

But I don't have a clear idea about what the best way would be to separate the two, so I'll try to address your comments in point form as I think there are a few things to consider:

• Firstly, I'm not currently aware of any other ORMs that could be used to replace the clorm factbase/querying stuff.

  • Most of the well-known ORMs are all about connecting to database systems (eg, Oracle, Postgres, MySQL, SQLite, etc) whereas clorm is about querying an in-memory python data structure. So I think they address different use cases.
  • There are some (or at least one) other libraries that are closer to clorm. Doing a quick search I came across these two projects:
    • https://github.com/ptmcg/littletable looks like the closest to clorm's querying stuff. It would be worthwhile looking more closely at this.
    • https://pydblite.readthedocs.io/en/latest/pythonengine.html
  • If you find some other candidate library let me know and I'll add it to the list.
  • One aspect of Clorm that I think is important is that it doesn't reduce the relational system to a set of flat table-like structures (ie row/column). So you can specify a where clause with sub-fields such as where(P.a.b.c.d == 5). AFAIK you can't do this with normal SQL. I think this is really useful for mapping to ASP facts with complex terms, so I think any replacement system would need to have the features to support this.
  • I think the flexibility of being able to address sub-fields would rule out most SQL based libraries. So for example, SQLite has an -in-memory mode but it is still SQL.

• So, secondly, unless we find some existing ORM that we can use then it comes down to seeing if it is possible to separate these two aspects of clorm. My main thought here would be to start a new project that looks at how to provide a factbase/query mechanism for something like Python dataclasses.

  • It would need some way to generate the field path objects (e.g. the P.a.b.c.d in a query P.a.b.c.d == 5). Could possibly have a class decorator for a dataclass that generates them.
    • We could start with Clorm's implementation for the underlying query mechanism, but it could do with some refactoring anyway.
    • Once that is finished then look at what are the extra bits needed to map these objects to clingo facts.

Anyway, hopefully the above makes some sense.

[MaxOstrowski]

Your comments actually sound awesome and make perfectly sense. I am not very familiar with most ORM systems, and the nested structure you describe is definitely a plus. I think a lot of ORM's support any database, including in-memory ones, so this shouldn't be a problem. The advantage of an external database system would of course be speed and maintenance, as you do not have to write all the query stuff again - and there is a lot of sql trickery out there. In a perfect world it would be possible to map the field path objects down to flat tables efficiently. I do not know if this is possible at all. If yes, using an existing system might make sense. Otherwise I like your idea of splitting clorm into the factbase/query stuff and the conversion to clingo facts. But I think thinking about the extra bits to map whatever objects to clingo facts should be the first step. If this can't be done efficiently, then the only advantage of this whole rewriting would be for projects where you use clorm but not clingo (as you said. I've done the same :smile: ).