damavis / airflow-hop-plugin

Apache Hop plugin for Apache Airflow - Orquestate Apache Hop pipelines and workflows from Airflow
Apache License 2.0
11 stars 6 forks source link

Tasks #32

Open pauloricardoferreira opened 9 months ago

pauloricardoferreira commented 9 months ago

hello... I'll give you some improvements soon... I'm going to fork and then make a PR. thanks

pauloricardoferreira commented 9 months ago

I have evolved the project a lot.

I made some improvements that the project needed.

As Apache Hop was developed to work in a decoupled way, I made some adjustments to make the most of the Plugin.

At the moment I am creating a routine to create a json with the metadata.json that the plugin needs, this way, it will no longer be necessary to export through the graphical interface.

As soon as possible, I will create a PR and hope it is reviewed and approved.

Thanks.

PereAL7 commented 9 months ago

Hello, I've seen your PRs, I will be checking and testing the changes in the next couple of days. I'll keep you informed.

pauloricardoferreira commented 9 months ago

TKS.

Something interesting is that the Plugin reads the variables used in the project.

And one of the important variables for the Project is PROJECT_HOME, which needs to be defined in the Apache Hop interface for perfect execution.

image

Example project configuration.

It is necessary to create this environment variable, pointing to the exact location of the project.

This way the plugin works in an exemplary way.


You could also find a way to make this variable and add it to the Payload XML.

It is found in the hop configuration file.

"projectConfigurations" : [ { "projectName" : "hop_repo", "projectHome" : "/opt/projetos/hop_repo", "configFilename" : "project-config.json" }

I didn't implement this part, but I believe it is possible.

This way the plugin is completely independent

TKS

pauloricardoferreira commented 9 months ago

I did a test running only the Pipeline, and the process ran perfectly.

Now, when a Workflow calls a pipeline, it cannot execute it, because it cannot find the PROJECT_HOME path, and therefore cannot find the pipeline path.

I tried to find it in the code, but I couldn't

pauloricardoferreira commented 9 months ago

TKS.

Something interesting is that the Plugin reads the variables used in the project.

And one of the important variables for the Project is PROJECT_HOME, which needs to be defined in the Apache Hop interface for perfect execution.

image

Example project configuration.

It is necessary to create this environment variable, pointing to the exact location of the project.

This way the plugin works in an exemplary way.

You could also find a way to make this variable and add it to the Payload XML.

It is found in the hop configuration file.

"projectConfigurations" : [ { "projectName" : "hop_repo", "projectHome" : "/opt/projetos/hop_repo", "configFilename" : "project-config.json" }

I didn't implement this part, but I believe it is possible.

This way the plugin is completely independent

TKS

I encountered the problem with the PROJECT HOME variable.

I needed to add a variable to root: root.append(project_home).

PR Created.

pauloricardoferreira commented 9 months ago

hello... Did you test the plugin?

pauloricardoferreira commented 7 months ago

hello... Did you test the plugin?

PereAL7 commented 5 months ago

Hello, sorry for taking so long. Life got busy. Could you unify all your PRs into a single PR? It is really bothersome to perform testing this way and I don't like merging partial implementations. Also, I've noticed some breaking changes. Please, try to maintain backwards compatibility. If you need to you can create a new "updated" operator.