[Solr Load] Change Solr indexing strategy

[ficolo]

• [ ] Create a luigi task to setup an on demand Solr Server on the cluster

  • Check if the workdir exists
  • Check if the workdir has data
  • Clean up workdir data
  • Copy workdir template if it doesn't exists
  • Load the right Java version

• [x] Configure the on demand Solr Server using DEV Solr config

• [ ] (Optional) Make the on demand Solr Server to create onely one segment: https://stackoverflow.com/questions/37884217/how-to-force-solr-6-to-merge-segments

• [ ] Change the parquet2solr tasks to simple tasks posting JSON documents to the Solr Server

  • Dump parquet as JSON
  • Read JSON
  • Check Solr core Schema to determine allowed fields
  • Check Solr core schema to cast numeric types
  • Index files in parallel
  • Optimize the core after the Indexing is done

• [ ] Delete the tasks using the old solr indexing strategy