search

weso / sparkwdsub

Spark processing of wikidata subsets
MIT License
0 stars 3 forks source link

Merge branch spark-cluster-works with master #20

Open labra opened 2 years ago

labra commented 2 years ago

During the tests to make sparkwdsub run on AWS, @thewillyhuman created a branch called thewillyhuman/spark-cluster-works which works on AWS because it doesn't use decline and ensures that the dependencies use the right versions.

At the same time @labra kept working adding features to the master branch and at this moment the situation is that when we need to deploy on AWS or run the system, we do a manual process to integrate the latest changes in the master branch with the other branch.

Most of the changes are quite easy, as the main thing that changed was the precisely the Main code that uses scallop instead of decline to handle command line options. Anyway, I think it deserves to merge both branchs and keep working on a unified one.

labra commented 2 years ago

Yesterday, I tried to compile the code on the thewillyhuman/spark-cluster-works branch and it failed...so I stopped...we have to ensure that it compiles before continuing.