Serialization API

[DhruvaSambrani]

Is your feature request related to a problem? Please describe. Some Julia types cannot be directly represented in SearchLight. One example is enums. These can however be converted to a DB safe type like ints or strings. However, This would imply that the Julia type needs to be changed to reflect this.

Describe the solution you'd like A transformation should be applied from Julia to DB and DB to Julia. This can be simply done by writing a function transform_to_db(<:AbstractModel) = identity and transform_from_db(<:AbstractModel) = identity. The user can then overload these functions for their Types if necessary.

Describe alternatives you've considered Workaround: Save the julia types as db safe types, and write call functions which transform the type as necessary to get the required type.

[DhruvaSambrani]

Or you could add a column specific transformations too, but I'm not sure how that is implemented

[FrankUrbach]

Hallo Dhruva Because I'm write some code to extent SearchLight for my needs I can imagine what you are trying to accomplish. There are so many different types which haven't a possible datatype in databases that makes it near impossible to implement all besides the possibility to create custom structs. I've made forks of SearchLight, SearchLightPostgeSQL and SearchLightMySQL where I implemented the possibility to note the fields which will be stored in the database. This leads to the possibility to have intern fields which are not stored in the database. But this is only possible in the branch savingWithoutInterns. Be warned. This is not an official release and under development. I want discuss the implementation with Adrian (owner of SearchLight). If he decides to integrate the work in his repository it will be offical if not I will integrate the branch in my master branch. In the tests of the SearchLightPostgreSQL.jl you can read the test_models.jl and test_posrgresql.jl. They show, how these things are meant to be. If you will go further with the official brach a possibility could be to implement another struct with the fields you want, create a AbstractModel struct with the fields you want store and use the event :before_save the transfer the informations from one one struct to the AbstractModel struct with the logic you want. Feel free to ask any questions anytime. Best Frank

[DhruvaSambrani]

But in my way Package maintainers wouldn't need to implement the transformations. There will be absolutely no change in the way things are stored either. All that needs to be done is for the user to overload/specialize a transformation function which the package applies. If the user hasn't implemented this, then it will fall back to the default identity transformation and Multiple Dispatch will handle the rest.

For example, while retrieving from the db, all calls finally lead to this function https://github.com/GenieFramework/SearchLight.jl/blob/d6984180f0bea94dfdde83dc2d4092c88139fa7c/src/SearchLight.jl#L344

Now we just add a line

transformed_row = transform_from_db(_m, row)

And just use transformed_row instead of row everywhere. Further, add a function definition

transform_for_model(model::AbstractModel, df::DataFrames.DataFrameRow) = df

And everything JustWorks(TM)

When the user wants to transform the df, they specialize to

function transform_for_model(model::MyConcreteModel, df::DataFrames.DataFrameRow)
    # Some transformations of df
    df[:num] = -df[:num] # not sure if this works, but I'm trying to negate the column
    df
end

[DhruvaSambrani]

Note that I'm sending an object, and not the type, because Julia can specialize on type, not on value, and I don't know if there is anyway to specialize on value.

[FrankUrbach]

It would be possible to do so but in my humble opinion this is much to deep in the internals of SearchLight. If somebody use this technic and an error occur hunting this down to the user function would be very difficult. Also I wouldn't mention to deal with Dataframes on the user side of the framework. I would prefer to deal with the objects and this is possible if you look in my fork in the branch I mentioned above. In the final instance, Adrian should decide how he wants to handle it. Just my 2 cents.

[essenciary]

SearchLight provides the Serializable module especially for this and the default implementation is the JSON serializer. It used to work in the older API (where we were using special _fields) but needs to be reimplemented to work with the new API. I'm thinking something in the line of a Serializable wrapper type plus serialize and deserialize methods. And the users can define custom/additional serializers.

[essenciary]

The new Serializable API has been implemented for a while, closing this (API needs to be documented).