Add experimental custom autocompletion

[sorhawell]

Close #580

• [x] implement custom Rstudio code completion
• [ ] native R / radian code completion, to be as good as Rstudio completion or what possible.
• [ ] support vscode-R completion via their webserver feature
• [x] add pl$polars_code_completion()

also to support completion:

• added $columns()-method and names() for GroupBy and LazyGroupBy
• improved "method not found" error message
• added missing .DollarNames.RLazyGroupBy method
• deprecated pl$extra_auto_completion() the old way to activate native R completion.

[sorhawell]

have made some progress. It is a bit slow as these Rstudio internals seemingly have no public documentation

the look up help box is for the wrong function, but looks fine

[sorhawell]

completions for chains, icons shows if method / sub-namespace ("list")

... and assignment (columns<-)

Auto completions can be used to "tab-start" an expression from a column name

works for lazy too

[sorhawell]

interactive syntax explaining via messages. In ipython-polars I miss this feature. Sometimes I tab-complete and nothing happens and I don't know why. Most often because some earlier error in expression.

alias is missing an argument

[sorhawell]

@eitsupi @etiennebacher

It is possible to make very sweet Rstudio auto completion. I would be possible to also add interactive help-docs or table views of DataFrame.

So far only data.table, dplyr and reticulate and maybe 3 other packages have their completions in the rstudio-source code. At some point we may apply to get our completions included. I guess it will be denied as we use full evaluation of the entire syntax which could slow down the IDE. I have a 3 seconds timeout to block any unintended eager evaluation. R timeout cannot block rust code, before returning to R.

Here I make use the completions methods can be wrapped with custom handlers and injected into the rstudio ide environment. In theory multiple packages could all wrap the same rstudio completion without conflicting.

This approach is using the private api of Rstudio-api, but judging from the blame history. This code part is quite stable on a yearly basis.

[sorhawell]

to activate auto-completion run this

source("~/Documents/projs/r-polars/R/rstudio_completion.R")
.dev$activate_polars_rstudio_completion()
library(polars)

the weird .dev-prefix would not stay. It is just a convenient way to update completion functions without restarting session or building polars package to do faster "trial and error".

[sorhawell]

We also have some reasonable code completion for native R terminals which could be further refined. In the past I was discouraged how bad the native R utils completion worked combined with Rstudio IDE. But that is no longer a problem with this PR. We can detect what IDE is being used.

[etiennebacher]

Love this, the examples look amazing! I'll try a bit more later.

[etiennebacher]

That would be cool to have in the next release (not before a few weeks I think). @sorhawell if you're short on time, maybe you can let aside the radian/VS Code objectives and address them in another PR later?

I also think this kind of completion could be enabled by default in RStudio and in the terminal using .Platform$GUI %in% c("RStudio", "RTerm") in .onLoad. What do you think?

[eitsupi]

I also think this kind of completion could be enabled by default in RStudio and in the terminal using .Platform$GUI %in% c("RStudio", "RTerm") in .onLoad. What do you think?

I'm not familiar with these, but perhaps it can be enabled for interactive sessions only?

[etiennebacher]

This is what rstudioapi uses to detect whether RStudio is used:

rstudioapi::isAvailable
#> function (version_needed = NULL, child_ok = FALSE) 
#> {
#>     if (child_ok && isJob()) 
#>         return(callRemote(sys.call(), parent.frame()))
#>     identical(.Platform$GUI, "RStudio") && version_ok(version_needed)
#> }
#> <bytecode: 0x00000285110f1b98>
#> <environment: namespace:rstudioapi>

[sorhawell]

without rstudioapi it is possible to check for if it is an rstudio session like this

"tools:rstudio" %in% search()

that is the internal rstudio API which is being patched by the code completion. I think it should always be actively opt-in rstudio-completion, because patching another API at run-time is brute manners.

[etiennebacher]

I think it should always be actively opt-in rstudio-completion, because patching another API at run-time is brute manners.

Are there some clear downsides in hacking this API? Is the autocompletion for other packages affected by those changes?

I would be fine with keeping it experimental and opt-in for now (maybe with a message when loading the package?) but I mean at some point we should enable it by default as it changes drastically the experience with polars.

[eitsupi]

Any update on this?

@etiennebacher Are you willing to take this on? I am not very interested in this as I do not use RStudio and have no motivation to test it.

[etiennebacher]

@etiennebacher Are you willing to take this on?

Yes I'll finish this one

[etiennebacher]

I don't know how to test completion in "native R". I ran vanilla R in bash and in Powershell and couldn't get any completion (for any function, not just polars)

[etiennebacher]

I am wondering if there is any worth in the option to enable/disable this.

I wanted to enable this automatically at first, but I'm not confident on how it will interact with other packages that might add their own custom completion (also @sorhawell wrote most of this, I'm not so familiar with the autocompletion internals). I'd rather keep this experimental and opt-in for now

[sorhawell]

@eitsupi first time using task to update deps and build. That was a very out-of-the-box experience. Nice!

@etiennebacher native R terminal has no drop down for multiple suggestions, and there is something annoying where it will delete what alraady has been written. Right now it works best for the very final completion.

> library(polars)
> pl$code_completion()
Using code completion in 'native' mode.
> # writes this and tab:  pl$col()$ab
> pl$col()$abs() # completes s()
polars Expr: col("").abs()
> rc.status()$comps # use rc.status()$comps for unit testing
[1] "$abs()"
>

it works much better with the radian terminal

writes this and tab: pl$col()$a

[sorhawell]

I found this for unit testing of native-native + native-radian

> utils:::.assignToken("pl$col()$a")
> utils:::.completeToken()
> utils:::.retrieveCompletions()
 [1] "pl$col()$abs()"             "pl$col()$add()"             "pl$col()$agg_groups()"      "pl$col()$alias()"          
 [5] "pl$col()$all()"             "pl$col()$and()"             "pl$col()$any()"             "pl$col()$append()"         
 [9] "pl$col()$approx_n_unique()" "pl$col()$arccos()"          "pl$col()$arccosh()"         "pl$col()$arcsin()"         
[13] "pl$col()$arcsinh()"         "pl$col()$arctan()"          "pl$col()$arctanh()"         "pl$col()$arg_max()"        
[17] "pl$col()$arg_min()"         "pl$col()$arg_sort()"        "pl$col()$arg_unique()"      "pl$col()$argsort()"        
[21] "pl$col()$arr()"