Research: Configure full importOssDirectory Pipeline in Data Warehouse

[ravenac95]

Currently this is a research task due to some complications in actually attempting to implement this using the assumed-to-work pipeline of tools (airbyte, dagster, dbt) see here. There's a now deprecated integration that made some of this work. Some additional planning needs to be made in order to get this to properly function. Once that's complete we can/should complete the documentation related to the architecture and the developer docs as we hope to see it. However, as it is, the current limitations add some complications that would complicate collaborations as the number necessary of tools increases.

This research should be time boxed for another day or so.

[ravenac95]

Ok spent the majority of today attempting to research alternatives. First, this issue wasn't clear as to why we would want any of this behavior. The main thing that we should be looking to to design is an easy to collaborate as-code data collaboration platform. However, what I found in attempting to go from using the UI to turning this into code is that not all the tools have a way to properly configure everything as code. Namely, this issue exists between dagster and airbyte. Airbyte is seemingly going through a massive refactoring of it's source/destination/connector management that will either break (it's not yet clear if it will break) features that we could use to fill the gaps that the current tools deprecate or it will address all associated problems. However, things are currently half-baked at airbyte and so it's a bit of a risk. As it stands, it's not possible for us to write custom connectors for airbyte and also define those sources/destinations as code without writing our own integration. I wouldn't mind writing that integration if the API we would need to depend on wasn't being "removed" in early 2024

That being said, unlike the under-engineering from the previous iteration of OSO I'd like to make sure we've looked at as much of what is actually available in this space. I've considered the following options

• Write our own dagster + airbyte integration, relying on the Configuration API which will be "disappearing" but they have noted that they will be replicating more API endpoints to the new API (which currently doesn't support what's needed).
• Explore a different connector system:
  • This isn't ideal as airbyte seems to be the easiest to use of the open source connectors. However, it does require the UI and configuration is not so flexible. So if we want to change course on this, now is the time.
  • Alternatives
    • CloudQuery - This is the main contender in my opinion. It is also mostly language agnostic but it does require the use of protobufs to create a connector.
      • Pros:
        • Language agnostic
        • Easily configured as code
        • Execution is simple
      • Cons
        • protobufs might make some things annoying for other devs
    • Singer - Singer has quite a bit of connectors already. Airbyte had some negatives to say about this here.
      • Pros:
        • Language agnostic
        • When used with something like meltano it seems it seems to be easily configured as code
      • Cons
        • Apparently lots of broken existing taps not certain this will affect us.
    • Sling - Not much on this, the project is relatively new but it seems it's officially supported through dagster. See here
• Explore an alternative to dagster
  • This won't entirely solve the problem it seems but it could be necessary.
• Changing how we think about writing some API integrations
  • Doing this doesn't necessarily fix some of the issues with defining data ingestion as code, but this could alleviate some development impediments that we're currently experiencing. The current design we have is that some inputs from the DB are used as ways to make conditional API requests. For example, oss-directory lists github repo/organization urls for a given project. From there, we need to resolve all of these github urls into their downstream artifacts. We currently do this by querying the API. If we instead always queried all of github through the github archive this might not be necessary and we can do everything through dbt. However, this is only one example that has a data pipeline where the output of one step in the pipeline guides the inputs to another (that isn't in some kind of DB). If we need to integrate something that does need to make API calls then we would still run into many of the same problems.

[ravenac95]

Completed and merged