[WIP]: Parser

[twinstar26]

ToDo list:-

• [ ] Add token more rules. (I have added basic rules. More rules will be added after discussion)
• [ ] Add grammar rules
• [ ] Add code for generation of token_list each time clai starts (redirect output of copmgen -c)
• [ ] Integrate parser into clai

[twinstar26]

Which additional tokens to pick? (I think we should at least pick all the non alphabetic chars. We will need some of them while defining grammar.)

We can create infinite re to pick tokens but thats not possible. Hence we use ml/dl. For now i am just picking commands and flags. Rest everything will be ignored and will be marked sensitive.

[twinstar26]

Lex and yacc pip modules are not updated regularly. Hence I am taking them directly from source. Source is more compatible with python 3.6 which we are using.

Final commit will pull directly from source so that we dont have to maintain/sync upstream regularly.

[TathagataChakraborti]

Thanks @twinstar26 for opening the issue 👍 Can you chalk out the use cases, i.e. with examples of things you are trying to achieve, in the corresponding issue: iss28. We can discuss them further, either here on in Slack, then. Thanks!!

[twinstar26]

I think all discussions must be done here in Github. Its easy to reference them in future. All long discussions/debates in slack.

I think that for simple tasks like determining which command is being issued, etc, we should use parsers and not rely on ML/DL models as parsers are deterministic in nature while ML/DL are probabilistic. If I am standing at a node and I have to make a decision but the branching factor is greater than a predefined N, then instead of trying to solve entire tree which is computationally heavy we can turn to ML/DL models.

That being said we will know which command is being issued, flags, arguments even in complex cases like pipes, sub-commands (bash) by defining appropriate grammar. So why broadcast queries to all the plugins when we can just send to appropriate plugins. This will further decrease execution time of clai.

For user privacy I think its best that we forward queries to plugins where only command names and flags should be visible. Rest everything will be marked private. In future if some plugins need extra data then we can provide them after taking user permission (during installation of that plugin). But outputs of commands are again passed through clai for obvious reasons. There we might face difficulty in hiding sensitive information as structure of output will be different for different commands and we can write only so much grammar rules but we can figure that out afterwards.

Lastly to make clai more useful, more appealing, more faster, then, all commands should not go through clai. For example, I am bash expert and I dont need clai to process every single command thereby decreasing my workflow speed, but I also get stuck sometimes so instead of issuing clai start > my_command > clai stop, it would be useful if clai had a call-sign (like how does google assistant has, "ok google") so everything will come under 1 command. This detection of call-sign can be covered in parser.

Please leave your thoughts... :)

[TathagataChakraborti]

Hi @twinstar26 thanks for the comments.

I think that for simple tasks like determining which command is being issued, etc, we should use parsers ..

I agree. This is a great parser to look at: explainshell. It's open source as well.

.... why broadcast queries to all the plugins when we can just send to appropriate plugins. This will further decrease execution time of clai.

In principle there are trade-offs between prior and posterior orchestration: see here. This is primarily to keep the skills layer and the orchestration layer separate in design. But yes, I agree we can do better than "broadcast all", either through learning based orchestration patterns or by asking skills to be specified with features that can be used at the orchestration level and using those for prior determination.

Though I should point out that all the skills are executed in separate threads. So theoretically, prior orchestration does not impact execution time. Of course, practically, there are other kinds of system loads which definitely come into consideration for prior versus posterior orchestration.

For user privacy I think its best that we forward queries to plugins where only command names and flags should be visible. Rest everything will be marked private.

This is a great idea 💡 This would also likely help skills to focus on important parts of the command text versus those that are user specific.

Lastly to make clai more useful, more appealing, more faster, then, all commands should not go through clai. For example, I am bash expert and I dont need clai to process every single command thereby decreasing my workflow speed, but I also get stuck sometimes so instead of issuing clai start > my_command > clai stop, it would be useful if clai had a call-sign (like how does google assistant has, "ok google") so everything will come under 1 command.

This is correct. With CLAI we made the most generic framework, as a platform where you can build any use case you want. For deployment to end users, there is a lot of scope for optimization:

• You could suppress the broadcast to all, and just do call sign based invocation. This still means there will be a little lag since the socket and server are still active.

This is probably the best way to speed stuff up.

• You can suppress the pre-process step altogether, this means that the skills only activate after execution --> in this pattern, for natural language commands you will hit a bash: command not found first but then get the result as an attach to the stderr. This sort of setup will mean there is almost zero lag between command type in and execution, but of course this cuts out pre-execution user interaction patterns.

We would love contributions in service of speeding up CLAI that enable these specific modes, for example. We have a catch all issue for this: iss32.

Btw, we do have the call-sign mode already (we just didn't kill the generic use case). Check out how to do it here under Interaction Patterns.

[TathagataChakraborti]

Archiving this until it becomes active again.