New Processor: Datetime to string

bossenti commented 9 months ago

Body

Description

A user should have the ability converta datetime values to a human-readable string value in a pipeline. Therefore, a new processor is required that provides the functionality to convert a datetime value to a string value based on a given string format.

Example Configuration

StringFormat : A format string that describes the format of the datetime string

The string format should be configurable with a conveniant UI dialog, ideally similar to our Notification sink:

Additional Functionality

If you have any other ideas for the functionality of this new processor, please feel free to add them here.

Mentoring

As this issue is marked as good first issue: one of @dominikriemer, @tenthe, or @bossenti are happy to provide help for getting started, just tag (one of) them if you want to start working on this issue and need some help. This issue is ideal for people that are interested in getting started with StreamPipes and its SDK.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

lab596 commented 9 months ago

Hey @bossenti, I am honestly pretty new to coding but I would love to contribute to this project if I got a chance. Taking a look at this issue, some of the concepts may be over my head but I would like to give it a try. For now I have cloned the repo however when I run the code I seem to be getting this message as I believe I have not set up my environment:

Checking if consul is available on host consul and port 8500
Could not connect to Consul instance...

I have taken a look at SteamPipes CLI however I seem to be stuck on the streampipes in the CLI. Whenever I try to run a command I see bash: streampipes: command not found but I'm not sure what I'm missing.

lab596 commented 9 months ago

Hey, I believe I found the hiccup, in the documentation it states to utilize the ./streampipes command in bash to access the streampipes CLI, however this forgets to take into account the file path, Instead the env was able to be initiated correctly using ./installer/cli/streampipes. Please let me know if this is accurate, I am able to update the documentation if needed.

lab596 commented 9 months ago

Whenever I try to set up an environment I am faced with the following error:

$ streampipes env -s pipeline-element
[INFO] Set environment to 'pipeline-element'
cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

@bossenti @dominikriemer @tenthe Any help is appreciated, thanks!

bossenti commented 9 months ago

Hi @lab596,

First of all, we really appreciate your willingness to contribute 🙂 Don't be afraid if you are not having that much experience, we are happy to assist you.

Please let me know if this is accurate, I am able to update the documentation if needed.

Yes, you are right. The CLI commands should be executed in the directory ./installer/cli/. If the documentation does not explain this properly, a PR is welcome. Please be aware that our documentation is maintained in a dedicated repository: https://github.com/apache/streampipes-website

bossenti commented 9 months ago

cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

Do you observe this error when executing the CLI command in ./installer/cli?

lab596 commented 9 months ago

Do you observe this error when executing the CLI command in ./installer/cli?

Yes, I have tried multiple methods including using the full command ./installer/cli/streampipes env --set pipeline-element as well as setting installer\cli as a windows path and running the command above: streampipes env -s pipeline-element. Both lead to the same error:

[INFO] Set environment to 'pipeline-element'
cp: target 'Projects/streampipes/installer/cli/.spenv' is not a directory

bossenti commented 9 months ago

I just gave it a shot on my Windows machine as well. On my end running ./installer/cli/streampipes env --set pipeline-element completes successfully with below output:

[INFO] Set environment to 'pipeline-element'

Looks like there is a hick-up with respect to the path where the config should be written to. Can you please execute echo $STREAMPIPES_WORKDIR and share the result? 🙂

lab596 commented 9 months ago

Please be aware that our documentation is maintained in a dedicated repository: https://github.com/apache/streampipes-website

@bossenti, I have created a PR in that repo with a few minor changes please let me know if that correctly updates the documentation. Additionally, I believe that documentation also has to be updated on this repo here.

lab596 commented 9 months ago

Can you please execute echo $STREAMPIPES_WORKDIR and share the result? 🙂

I simply get an empty line:

$ echo $STREAMPIPES_WORKDIR

bossenti commented 9 months ago

Okay, that's expected, We just set it while the script is executed I was just wondering if this variable is somehow overwritten.

Since I'm running out of ideas, maybe @dominikriemer @tenthe or @RobertIndie are able to support here

lab596 commented 9 months ago

Okay, thanks for the update.

Actually, I'm not sure if I need this environment. Can someone please clarify? This is the original error I was faced with:

2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Checking if consul is available on host consul and port 8500
2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Could not connect to Consul instance...
2023-09-09T10:24:16.866-07:00  INFO   --- [           main] o.a.s.s.consul.ConsulProvider            : Retrying in 1 second

I just originally assumed that is because of the environment setup. Any additional info is appreciated.

dominikriemer commented 9 months ago

@lab596 This error typically occurs when you are running an extensions service locally, while Consul (and other services) run in a container. You can set two environment variables SP_DEBUG=true and SP_CONSUL_HOST=localhost to resolve this issue for local development.

lab596 commented 9 months ago

Hey, I believe I tried this by running this in the terminal. However, nothing happened. If this is not what you meant can you please elaborate?

export SP_DEBUG=true
export SP_CONSUL_HOST=localhost

bossenti commented 9 months ago

@lab596 to better support you here, I'd like to ask what you are trying to achieve. Do you just want to start StreamPipes to try it out? Or, do you want to get the development setup of StreamPipes ready on your machine?

lab596 commented 9 months ago

@bossenti I have successfully started and explored StreamPipes by following the basic installation instructions here.

However, the problems I am facing is when trying to deploy a development setup by following the instructions here and by extension the CLI setup as outlined here.

dominikriemer commented 9 months ago

@lab596 thanks for clarifying! The setup should work out of the box - here is a short description what I do when developing a pipeline element:

Stop the docker-compose instance of the basic installation (you might also consider to clean this instance by executing docker-compose down -v.
To set up the development environment, go to installer/cli, configure a template (./streampipes env --set pipeline-element) and run ./streampipes up -d.
You should then have a running StreamPipes instance including the UI with backend ports which are configured for local development.
When you open the IDE of your choice, create a new pipeline element and execute the Init method (e.g., try with the extensions-all-jvm module. Here is the place where you need to configure the environment variables as startup env variables (SP_DEBUG and SP_CONSUL_HOST). This will internally configure the extensions service for local development.
If you have any problems, you can check the logs of the extensions service (there should be a log message which shows where the system tries to connect to the core to register pipeline elements).

Hope this helps! Please ask if anything is unclear.

lab596 commented 9 months ago

Hey @dominikriemer,

Thanks for all the help. I was able to set up most of the development setup using your instructions however when attempting to create a new pipeline element to test it out I was facing this error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/streampipes/connect/iiot/ConnectAdapterIiotInit
        at org.apache.streampipes.extensions.all.jvm.AllExtensionsInit.provideServiceDefinition(AllExtensionsInit.java:48)       
        at org.apache.streampipes.service.extensions.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:51)
        at org.apache.streampipes.extensions.all.jvm.AllExtensionsInit.main(AllExtensionsInit.java:40)
Caused by: java.lang.ClassNotFoundException: org.apache.streampipes.connect.iiot.ConnectAdapterIiotInit
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
        ... 3 more

What I first did was ensure to add the startup env variables by running (I'm trying to run extensions-all-jvm as suggested):

$ docker run -e SP_DEBUG=true -e SP_CONSUL_HOST=localhost -d apachestreampipes/extensions-all-jvm:0.93.0-SNAPSHOT
701e4b3b072da950b980bf2bd893365912bb99fe4e51bc963d97a1323a08ee12

Then I attempted to run the AllExtensionsInit.java within the streampipes-extensions-all-jvm folder.

Am I missing something?

bossenti commented 8 months ago

@dominikriemer

apache / streampipes