Feature: replace panic message "tried to run FaninNode without hooking the in channel up to another node" into a nicer error message

alarbada commented 3 months ago

Feature description

Related to #1026, I get the same panic:

         `::::::::::‘        Conduit v0.10.0 linux/amd64
             ‘‘‘‘            
2024-05-30T11:45:17+00:00 INF All 0 tables opened in 0s component=badger.DB
2024-05-30T11:45:17+00:00 INF Discard stats nextEmptySlot: 0 component=badger.DB
2024-05-30T11:45:17+00:00 INF Set nextTxnTs to 0 component=badger.DB
2024-05-30T11:45:17+00:00 INF loading processor plugins from directory /home/guillem/projects/conduit-connector-pinecone/processors ... component=plugin.processor.standalone.Registry
2024-05-30T11:45:17+00:00 WRN could not read processor plugin directory error="open /home/guillem/projects/conduit-connector-pinecone/processors: no such file or directory" component=plugin.processor.standalone.Registry
2024-05-30T11:45:17+00:00 INF standalone processor plugins initialized component=plugin.processor.standalone.Registry count=0 plugin_path=/home/guillem/projects/conduit-connector-pinecone/processors
2024-05-30T11:45:17+00:00 INF builtin plugins initialized component=builtin.Registry count=16
2024-05-30T11:45:17+00:00 INF builtin plugins initialized component=plugin.connector.builtin.Registry count=6
2024-05-30T11:45:17+00:00 WRN could not read connector plugin directory error="open /home/guillem/projects/conduit-connector-pinecone/connectors: no such file or directory" component=plugin.connector.standalone.Registry
2024-05-30T11:45:17+00:00 INF standalone connector plugins initialized component=plugin.connector.standalone.Registry count=0 plugin_path=/home/guillem/projects/conduit-connector-pinecone/connectors
2024-05-30T11:45:17+00:00 INF processors initialized component=processor.Service count=0
2024-05-30T11:45:17+00:00 INF connectors initialized component=connector.Service count=0
2024-05-30T11:45:17+00:00 INF pipelines initialized component=pipeline.Service count=0
2024-05-30T11:45:17+00:00 INF pipeline started component=pipeline.Service pipeline_id=file-to-pinecone
2024-05-30T11:45:17+00:00 INF pipeline configs provisioned component=provisioning.Service created=["file-to-pinecone"] deleted=[] pipelines_path=pipeline.destination.yml
2024-05-30T11:45:17+00:00 INF grpc server started address=[::]:8084
panic: tried to run FaninNode without hooking the in channel up to another node

goroutine 75 [running]:
github.com/conduitio/conduit/pkg/pipeline/stream.(*FaninNode).Run(0x35d10d8?, {0x35e1710?, 0xc0000e68c0?})
        /home/guillem/go/pkg/mod/github.com/conduitio/conduit@v0.10.0/pkg/pipeline/stream/fanin.go:38 +0x22a
github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func2()
        /home/guillem/go/pkg/mod/github.com/conduitio/conduit@v0.10.0/pkg/pipeline/lifecycle.go:572 +0x227
gopkg.in/tomb%2ev2.(*Tomb).run(0xc00012cd70, 0x1fdb00001fd6?)
        /home/guillem/go/pkg/mod/gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /home/guillem/go/pkg/mod/gopkg.in/tomb.v2@v2.0.0-20161208151619-d5d1b5820637/tomb.go:159 +0xe5

Steps to reproduce

With the following pipeline.destination.yml

version: 2.2
pipelines:
  - id: file-to-pinecone
    status: running
    connectors:
      - id: file.out
        type: destination
        plugin: builtin:file
        name: file-destination
        settings:
          path: ./file.out
          sdk.record.format: template
          sdk.record.format.options: '{{ printf "%s" .Payload.After }}'

And the following command:

$ conduit -pipelines.path pipeline.destination.yml

I get the same panic. I did not need to build the pinecone connector.

Version

v0.10.0 linux/amd64

hariso commented 3 months ago

The internal FaninNode collects records from 1 or more sources and then forwards them to the next node.

Your pipeline has no source connectors, so the FaninNode complains that the in channel hasn't been hooked up to another node, i.e. no input to the FaninNode. In this case, that's quite expected, so I don't think there's a real issue here. Let me know if you still have some concerns about this, otherwise, I think we can close this one.

alarbada commented 3 months ago

@hariso Hi there Yeah, it is really a misconfiguration, it doesn't really make sense. I found that error just tinkering around. However, I'd like to see a better error than a panic, I had no idea what a FaninNode was until I delved into the conduit codebase.

Looks like a good first issue! If you don't mind @simonl2002 after I get some free time from the connectors work I'd like to see if I can make conduit output a better error than this panic.

Otherwise feel free to close this @hariso.

hariso commented 3 months ago

@alarbada Good that we're on the same page about this. I think you can change this issue into a feature that prints a nicer error message. We can easily do some checks before attempting to build a pipeline.

ConduitIO / conduit