Meetings and agenda

[buddiex]

Decide on each component of the pipeline and how to implement

• Ingestion
• Queue
• Processing
• Storage
• Dashboard

Decide on infrastructure

• Local machine/one machine setup
• container -> docker compose , Kubernetes
• virtual machine -> vagrant
• deployment
  • one server deployment
  • Kubernetes deployment
  • Saas deployment

Define development process/workflow

• Track work with github issues in combination with trello/git kraken boards
• PR and code reviews
• Sprint/release cycle
• Use of agile tools,

Assing tasks

• babatunde: scrum master
• Oladapo : Scrum master-junior

[buddiex]

project management:

• Doja, Grandee, Baba-D

• Plan deadlines for the udemy course:

  • Get the status of each person on the course
  • help the person set a target for the week
  • enter the goal/target on agile board
  • every sunday people give a report of progress..

what next:

• setup repo as public.
• meeting btw project mgnt
• schedule sprint planning meeting. Tuesday/wednesday
• create task:
  • setup dev environment

[buddiex]

High level Tasks from our meeting

• Setup dev environment

  • Setup docker compose file and docker files, simple hello world
  • Health check dev setup in multiple system

• Write Kafka producer for twitter

  • aggregate tweets per continent, country dimension
  • aggregate tweets per harsh tag per continent, country

• Unittest Kafka producer for twitter

• Implement spark streaming processing (Kafka consumer)

  • Decide Elastic search schema

• Design dashboard elements

• Implement kibana dashboard

• Setup ci-cd pipeline to test hello world

  • setup git actions
  • setup prod infrastructure
  • Test by running unit test before merge to main.
  • Test deployment of main to prod