delta-io / delta-examples

Delta Lake examples
Apache License 2.0
208 stars 76 forks source link

adding 2.4rc1 and template for future RCs #28

Closed nkarpov closed 1 year ago

nkarpov commented 1 year ago

Adding ability/template to run release candidates.

This requires adding an Ivy settings file and modifying the environment yml file to use an alternate index for resolving the python dependency.

In a notebook you pass the Ivy settings file like so,

.config(
        "spark.jars.ivySettings",
        "../../ivy/2.4.0rc1.xml"
    )

We can keep this internal for now and add some documentation later if necessary in a separate PR? Let me know.