How do the IBF Pipelines work? ("IBF Pipeline 101")

[diderikvw]

As IBF System Develop I want to have a basic understanding of the IBF Pipelines So that I can support Phuoc where needed, maintaining the Pipelines and doing small changes and fixes, can be a "sparring partner" for Phuoc, and think together with him and others in the team, including the PO about optimal ways forward in changing and improving the pipelines and how they fit into the IBF System's Software Architecture.

This is a follow-up from: Update the architecture diagram #1584

Accompanying Post in Teams here.

Refinement 5 Nov 2024:

1. Plan a team session of 1 hour.
2. Phuoc does a "show&tell" of IBF Pipelines (infrastructure, code, processes, repos, ...) => Team only, not with Aki and/or Jacopo for this first session.
3. Q&A + discussion
4. Define follow-up items, which could be: show&tell by Aki and/or Jacopo for more details and knowledge transfer.

[diderikvw]

@BlaiseSelvan @p-phung @gulfaraz I created this Spike as follow-up.

[diderikvw]

First opportunity on a Tuesday I could find is 25 November. I sent a calendar invite. @p-phung please let me know if you want to prep this meeting together. Nothing more to do at the moment, moving to status Waiting.

[p-phung]

@diderikvw I would like to make use this meeting to discuss how data specialist role can be integrating in the IBF e.g as a bridge between the IBF team and data pipeline owners. Do you agree to include it there in the meeting?

[diderikvw]

@p-phung Yes, IMO no topic is off the table for this first meeting. It is very likely we will need to plan follow-up meetings. Also with people outside of the IBF System Team.

As a side note:

between the IBF team and data pipeline owners

We as IBF Team are the IBF Pipeline owners. We agreed on that several weeks ago when we aligned on the product and the team. But the differences in understanding about what this means in responsibilities, accountability and practical work, are what we a.o. will talk about.

[p-phung]

@diderikvw for this meeting, I would like to invite @aklilu to show us the data pipeline for typhoon in the Philippines. The idea is that we share knowledge about the pipeline with SW and prepare for December/January period when we have low capacity in data (between as Aki, Jacopo and I are all away). Additionally, I would like to make use of this meeting to revisit our discussion earlier about data pipeline knowledge and explore what works for us

@aklilu I have forwarded you the invite for this meeting at Tue 26/11/2024 14:00. Could you join?

[diderikvw]

@p-phung Thank you for bringing this to my attention. I do not want Aki to participate in this first meeting, but first get our bearings as team. Also see Refinement. The meeting is just 1 hour and we will not be able to cover any complex topic extensively, so follow-up meetings will be needed anyway. @aklilu , nothing personal toward you. I know you are very knowledgeable about the pipelines. We will follow up with many more hours of meetings to do knowledge transfer in the coming months.

Good to know that you, Jacopo and Aki are unavailable in December and January. Do you know if all three of you are away for the full 2 months? This indeed requires us to look at knowledge transfer of IBF Pipelines a different way.

cc @BlaiseSelvan