search

krishna0592 / SMP

Sales Management Platform
0 stars 0 forks source link

create a base structure of Ingestion project #3

Open krishna0592 opened 5 years ago

krishna0592 commented 5 years ago
  1. For logging use the compare the Airframe log vs slf4j vs log4j (https://github.com/wvlet/airframe/blob/master/docs/src/main/tut/docs/comparison.md)
  2. compare dataframe vs dataset approach and choose the best one
  3. use the property files and update the values across the code

List of technology used are

  1. spark / scala
  2. local; kafka, hdfs - streaming
  3. Hbase, cassandra, Mysql, Hive, Elastic search
  4. Amazon s3, Azure Blob, SFTP / FTP
  5. csv, txt, tsv, json, parquet, avro
krishna0592 commented 5 years ago

Added the logger class to handle the Basic, count, DBQuery and Exception log level