Array Operations on Lists

[datasage]

We've been starting to test deepstream in production, at least for a few customers. We are using a custom, PHP client, to do updates for a server side. This does work for the most part. But the subscription nature of updating the records does cause issues sometimes.

In this issue I want to discuss lists. In our use case, we have several static lists that we add and remove records from. We never get rid of these lists, we just maintain them.

In situations of concurrency, its possible that the update fails due to another client updating the list record at the same time. Right now our solution is to retry the update, but its inefficient.

So to solve this I am proposing some additions to the message specification that would allow array operations to be directly to update lists without needing to subscribe. In particular for us, being able to do push and removeElement would cover our cases, but i could see adding additional operations like pop and slice.

I would be interested in building this, but I am looking for thoughts as to whether or not others may find this useful.

[yasserf]

We are looking to add a concept of version number -1. What this will do is bypass the version conflict in record transition, load the last object, apply the patch and then save it with the record version + 1.

This will be useful for adding a HTTP endpoint which will allow post requests to update records without having to subscribe first.

Would that solution work for you? The problem is it doesn't exactly support array operations, and lists are essentially just records under the hood. Theres a whole seperate discussion of us making a LIST api, but that requires alot more work and depends greatly on the underlying tech used ( since we would want to retrieve only the last X records from a list using that API as opposed to records which returns the entire json object, and certain things like REDIS supports that, but others don't ).

[datasage]

That would help in another case, but not for lists. I do have a case where the server needs to update the entire object, and it would be easier to not have to subscribe first to do that. For my use case, that is not likely to conflict with a version_exists issue though. Maybe the version of -1 could also bypass any subscription check on the update.

Lists are different for me, in most cases i need to just update the list without reading it as all i need to do is remove or add an element. A list api operations could fix this without significantly changing how lists work.

I can do some research on what Redis, mongo and others support for list operations and propose and API to support it. More than likely we can support everything, it just that some work may have to be done in a connector vs in the storage/message layer itself.

In the meantime, i could add support for the -1 version. That would help simplify my client quite a bit on all other operations.

Using a websocket client in languages like PHP does work. Subscriptions do somewhat complicate things. I think adding more support for more direct operations would allow languages like PHP to work better and could provide the interface needed for a REST interface.

[yasserf]

If you could add the -1 version that would be awesome.

I can do some research on what Redis, mongo and others support for list operations and propose and API to support it. More than likely we can support everything, it just that some work may have to be done in a connector vs in the storage/message layer itself.

True, but the question is more of what should deepstream support over what can we get.

When designing an application it isn't unlikely that some lists can become hundreds/thousands elements long. An ideal list API would have a minimum of:

• pagination so that you can specify what window you want to view in the list
• the ability to send operations such as added, deleted and moved without having to send alot of information over the wire
• the ability to do merge conflicts ( pretty complicated with lists )

[datasage]

Are lists indexed only intended to be index records to other individual records? Or can they be used to store other types of data?

The usefulness of pagination is a bit limited without being able to filter, or sort by other record data. Getting a slice of the records, you would have to know what indexes you are already looking for. If record names are the only thing that should be part of a list, then there may be a way to tie this together.

Having the option to get subscriptions automatically for the result set could save some round trips instead of subscribing to all records individually.

Being able to broadcast list updates would be awesome, would limit the scope of data being sent to an event.

If records are intended to be the only content of lists, then automatically removing or adding an item to a list on record creation/deletion would be helpful.

Merging is why firebase avoids 0 indexed arrays, they use a different sort of unique key for all elements.

[yasserf]

Keeping this issue open as lists is the killer feature deepstream needs.

It’s also not the easiest to implement and hence will most likely not come to be. Here’s to wishful thinking.