search

activeviam / par-student-spark-atoti

Project with students from CentraleSupelec to explore Spark API in order to power atoti with Spark
1 stars 0 forks source link

Table with vectors on databrick cluster #20

Open arnaudframmery opened 2 years ago

arnaudframmery commented 2 years ago

For now just a little script which generates a table with 5 columns :

The number of rows for each category, the number of category and the size of the vector are customizable

The script name is vector_generator, here is the link : https://adb-7605807453605139.19.azuredatabricks.net/?o=7605807453605139#notebook/104419597217635/command/104419597217637

OPeyrusse commented 2 years ago

I already commented orally on the fact that the same vector was used for every row, while I would prefer it to be randomly generated for each row. Everything but that is good for me.