Run query on a separate thread

[komsit37]

so that the plugin won't blocks the whole editor when running long query

[pindash]

This is not quite the same but is related, we can reuse connection, instead of opening and closing on each query. Maybe this should be optional, but if you have load balancing, then not closing the connection every time would allow for sticky sessions on the q side. And this would also solve running on a separate thread since we can use python's await feature on the open handle.

[komsit37]

That's a nice suggestion. I'm not familiar with python's await feature. Do you have some ideas on how this should be implemented?

[pindash]

I will try to write something over the weekend: But in the meantime I can point you here: https://stackoverflow.com/questions/44606142/using-async-await-syntax-with-twisted-callbacks

Qpython uses twisted under the hood, which means we should be able to reuse the deffered callback mechanism. One of the reasons, I have not been able to quite create a pull request, is that I don't fully understand the event model in sublime. It seems every time the window becomes active the object is instantiated, (i'm judging by the init messages on the console every time I click into the window). This seems rather wasteful and bizzare. I tried not closing the connection and seeing if it would be reused, but because a new instance of the QCon class is created each time, it just started leaking connection handles.

The reason this is relevant, is that I need to have a reference to the object created, the next time the window is activated, otherwise any messages that come back will be inaccessible in the new context. The simplest notion of what I have in mind: if I have two tabs A,B and I issue a long query from A to host 'a' and then I work in tab B and issue a query to 'b', if b's query should only be shown while tab B is active and A's will be shown when A becomes active there should not be a race condition.

[komsit37]

Thanks for the link. I think you can try to create a new

I tried not closing the connection and seeing if it would be reused, but because a new instance of the QCon class is created each time, it just started leaking connection handles.

It's not ideal solution, but you can put persistent connection in static map variable. We can use view id as a key so we can associate one connection per tab. (I haven't tested this thought).

Another (better) idea is to implement this as ApplicationCommand or WindowCommand (which I think is reuse throughout application or window lifetime). And trigger the command via sublime Module run_command(string, <args>) or sublime.Window Class run_command. (again I haven't test it) see https://www.sublimetext.com/docs/3/api_reference.html#sublime

if I have two tabs A,B and I issue a long query from A to host 'a' and then I work in tab B and issue a query to 'b', if b's query should only be shown while tab B is active and A's will be shown when A becomes active there should not be a race condition.

This sounds good.

[danielkrizian]

Have you considered running query on separate https://github.com/KxSystems/kdb/blob/master/l64/qcon process ?

[komsit37]

@danielkrizian no, not really. If I understand your idea correctly, executing query on a new qcon process would still block the editor because the plugin code would still need to wait for the process to respond.

[danielkrizian]

@komsit37 I haven't tested if single qcon process would block the whole Sublime Text 3 also for other qcon processes. What I tested (and use) is launching many separate qcon processes, each attached to its own buffer within Emacs editor - sending query within one qcon instance does not block neither the Emacs editor as a whole nor other qcon instances. Not sure whether Sublime Text has concept of separate buffers like Emacs

[komsit37]

@danielkrizian sorry I forgot to reply. I didn't use qcon in this plugin so it would be difficult to change. P.S. your emacs setup looks very nice!

[komsit37]

I implemented this feature in https://github.com/komsit37/sublime-q/commit/2a4e866ecfb11d5b7642efa0ff901becb2068ba0

Now sublime text won't block while executing q statements. This update is in v0.3.0 which should be available via package-control in a few minutes.

The limitation is that the query is still executed one-by-one. For example, you cannot send querries to multiple q sessions simultaneously. This is because sublime-text only has one alternative thread. Multiple statements will be queued and executed one-by-one.

[pindash]

Thank you! this is working for me