mlverse / pysparklyr

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect
https://spark.posit.co/deployment/databricks-connect.html
Other
14 stars 3 forks source link

Add prompt if when using `version`, and do not have a Host (Databricks) #91

Closed edgararuiz closed 11 months ago

edgararuiz commented 11 months ago

If passing version to spark_connect(), and Host is missing (because of using alternate auth), prompt user for installation if the the environment is missing. Today it errors out:

Error in use_envname(method = method, version = version, envname = envname,  : 
  No viable Python Environment was identified for Databricks Connect
version 13.3
Run: `pysparklyr::install_databricks(version = "13.3")` to install.
Error in `use_envname()`:
Error in `use_envname()`:
! No viable Python Environment was identified for Databricks Connect
  version 13.3
Run: `pysparklyr::install_databricks(version = "13.3")` to install.
edgararuiz commented 11 months ago

This work in dev now:

> remotes::install_github("mlverse/pysparklyr")
Skipping install of 'pysparklyr' from a github remote, the SHA1 (dcc61bd1) has not changed since last install.
  Use `force = TRUE` to force installation
> packageVersion("pysparklyr")
[1] ‘0.1.2.9001’
> library(sparklyr)

Attaching package: ‘sparklyr’

The following object is masked from ‘package:stats’:

    filter

> sc <- spark_connect(
+   method = "databricks_connect",
+   cluster_id = "1026-175310-7cpsh3g8",
+   version = "14.1"
+ )
! No viable Python Environment was identified for Databricks Connect version 14.1
  Do you wish to install Databricks Connect version 14.1?

1: Yes
2: No
3: Cancel

Selection: