search

astrolabsoftware / spark3D

Spark extension for processing large-scale 3D data sets: Astrophysics, High Energy Physics, Meteorology, …
https://astrolabsoftware.github.io/spark3D/
Apache License 2.0
30 stars 16 forks source link

Scala reflection, Java (JVM), and python: they can live together. #90

Closed JulienPeloton closed 5 years ago

JulienPeloton commented 5 years ago

This PR adds new functionalities to pyspark3d by solving the problems of passing Scala generic arguments to python through py4j. The main thread of discussion is here. In short if a Scala method makes use of a ClassTag (i.e. does not explicitly mention the type of an argument), then you need to pass it explicitly in python:

myMethod[A : ClassTag](arg: A):

would be called in python (using py4j):

myMethod(arg, classtag(arg))

The new functionalities added in pyspark3d.spatialOperator are: windowQuery, KNN, KNNEfficient, CrossMatchCenter, and CrossMatchHealpixIndex (from the Scala package com.astrolabsoftware.spark3d.spatialOperator).

JulienPeloton commented 5 years ago

Btw, with this PR pyspark3d has all the features of spark3D. Youpi!

codecov-io commented 5 years ago

Codecov Report

Merging #90 into master will increase coverage by 0.23%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #90      +/-   ##
==========================================
+ Coverage   97.11%   97.35%   +0.23%     
==========================================
  Files          28       31       +3     
  Lines        1041     1133      +92     
  Branches      221      217       -4     
==========================================
+ Hits         1011     1103      +92     
  Misses         30       30
Flag Coverage Δ
#python 97.7% <ø> (+1.15%) :arrow_up:
#scala 97.24% <100%> (+0.01%) :arrow_up:
Impacted Files Coverage Δ
...in/scala/com/spark3d/geometryObjects/Point3D.scala 100% <ø> (ø) :arrow_up:
...main/scala/com/spark3d/python/PythonClassTag.scala 100% <100%> (ø)
...ala/com/spark3d/spatialOperator/SpatialQuery.scala 97.14% <100%> (+0.26%) :arrow_up:
geometryObjects.py 94.93% <0%> (-0.07%) :arrow_down:
spatial3DRDD.py 100% <0%> (ø) :arrow_up:
spatialOperator.py 100% <0%> (ø)
converters.py 100% <0%> (ø)
__init__.py 96.55% <0%> (+0.06%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f584007...32f946a. Read the comment docs.