Link - Githubissues

coatk1 / playground

testing and experimenting with everything

https://coatk1.github.io/playground/

MIT License

0 stars 1 forks source link

Link #93

Closed coatk1 closed 3 years ago

coatk1 commented 3 years ago

https://www.tutorialspoint.com/hive/hive_quick_guide.htm

coatk1 commented 3 years ago

Spark runs onto of Hadoop to make Hadoop faster
- MapReduce is a framework/ parallel programming to process data
- HDFS stores data in Hadoop
- Sqoop import/export data to/from HDFS and RDBMS
- Pig scripting to run MapReduce
- Hive converts SQL scripts to run MapReduce
- PySpark is Python wrapped around Spark
Data mining is to look for patterns and make predictions
Data-warehouse uses ETL
Data-lake uses ELT (all raw data)
Kafka is used for data streaming