Currently, we read the flows dir & save paths to flow files when the flow service starts up. When a message that needs to be transformed comes in, we import the flow file and run it. The import happens every time there's a message to be transformed, which seems inefficient.
We should investigate whether that import can be done once in the beginning when we read the flows dir. This requires any dependency in that flow file to be ready. E.g. if the flow file imports types from their moose sdk, then that sdk has to be generated already before we try to import flow files. This is likely much easier once data models start out in typescript rather than prisma models.
linear[bot]
commented
5 months ago
Currently, we read the flows dir & save paths to flow files when the flow service starts up. When a message that needs to be transformed comes in, we import the flow file and run it. The import happens every time there's a message to be transformed, which seems inefficient.
We should investigate whether that import can be done once in the beginning when we read the flows dir. This requires any dependency in that flow file to be ready. E.g. if the flow file imports types from their moose sdk, then that sdk has to be generated already before we try to import flow files. This is likely much easier once data models start out in typescript rather than prisma models.