-
We should start adding pyspark support.
kmike updated
5 years ago
-
While a proper implementation integrating at the JVM level (#541) will take time, we can create a quick prototype by wrapping pylance into PySpark functions.
Minimal scope:
* Read (no pushdown)
…
-
Need a new context (`:py`?)
Need to see how to integrate and use PySpark from scala.
-
ImportError: cannot import name 'crop_to_bbox' from 'acvl_utils.cropping_and_padding.bounding_boxes' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/acvl_utils/cropp…
-
I have installed pyspark in my single machine thru `conda install pyspark`
and I type `pyspark` in shell. But it can not launched and returns the error:
```
Could not find valid SPARK_HOME while se…
-
**Is your feature request related to a problem? Please describe.**
Hello, I see that this package supports Pandas, but does it support pyspark? I'd like to use this on large datasets and pandas is …
-
Would it be possible to share your pyspark implementation of these functions? I have seen that the full integration is planned for future updates, however you mention that you've ran the tests using S…
-
-
```python
import os
import subprocess
os.environ["JAVA_HOME"] = subprocess.check_output(["/usr/libexec/java_home", "-v", "1.8"]).decode().strip("\n")
import random
import pyspark
import findsp…
-
## Bug
#### Which Delta project/connector is this regarding?
- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)
### Describe the problem
Using latest py…