search

bsblabs / talend-job-exporter

Talend plugin allowing to use Talend TOS DI in command line to export jobs
Apache License 2.0
6 stars 14 forks source link

TOS 6.2.0 #4

Open alexis-gruet-deel opened 8 years ago

alexis-gruet-deel commented 8 years ago

Hi,

I successfully build w/ maven for TOS 6.2.0 by search and replace all occurrences of 6.1.1 in the POM.xml.

Also, according to the commit e583d0f7 i updated the src/main/resources/plugin.xml to match the right dependency :

<import plugin="org.talend.core.repository" version="6.2.0.20160510_1709"/>

However while running the command : 

./TOS_DI-macosx-cocoa -nosplash -data /Users/agruet/Downloads/TOS_DI-20160510_1709-V6.2.0/workspace -application com.bsb.tools.talend.export.JobExportApplication -targetFile /Users/agruet/TEST.ZIP -projectName KAAS_CAR_LEAD -jobsToExport "KT_API_TCP"  -context live

the log in terminal gives : 

[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.

But the ZIP File is not in the specified target folder.

Inside .metdata/.log i found : 

!SESSION 2016-06-15 18:36:00.266 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_60
java.vendor=Oracle Corporation
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_US
Framework arguments:  -application com.bsb.tools.talend.export.JobExportApplication -targetFile /Users/agruet/TEST.ZIP -projectName KAAS_CAR_LEAD -jobsToExport KT_API_TCP -context live
Command-line arguments:  -os macosx -ws cocoa -arch x86_64 -data /Users/agruet/Downloads/TOS_DI-20160510_1709-V6.2.0/workspace -application com.bsb.tools.talend.export.JobExportApplication -targetFile /Users/agruet/TEST.ZIP -projectName KAAS_CAR_LEAD -jobsToExport KT_API_TCP -context live

!ENTRY org.eclipse.compare.win32 4 0 2016-06-15 18:36:01.060
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.compare.win32 [84]
  Unresolved requirement: Require-Capability: eclipse.platform; filter:="(osgi.os=win32)"

        at org.eclipse.osgi.container.Module.start(Module.java:434)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1582)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1561)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1533)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1476)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)

!ENTRY org.eclipse.emf.ecore 2 0 2016-06-15 18:36:01.710
!MESSAGE Both 'org.talend.model' and 'org.talend.model' register a package for 'platform:/resource/org.talend.model/model/TalendFile.xsd'

!ENTRY org.eclipse.core.resources 2 10035 2016-06-15 18:36:01.807
!MESSAGE The workspace exited with unsaved changes in the previous session; refreshing workspace to recover changes.

!ENTRY org.talend.platform.logging 1 0 2016-06-15 18:36:03.292
!MESSAGE 2016-06-15 18:36:03,291 INFO  org.talend.components.api.context.osgi.GlobalContextOsgi  - Activated i18n messages (org.talend.components.api.i18n.osgi.I18nMessageProviderOsgi@13809363).

!ENTRY org.talend.platform.logging 1 0 2016-06-15 18:36:03.298
!MESSAGE 2016-06-15 18:36:03,298 INFO  org.talend.components.api.service.internal.osgi.ComponentServiceOsgi  - Registered the component: components#wizards#salesforce.edit(org.talend.components.salesforce.SalesforceConnectionEditWizardDefinition)

!ENTRY org.talend.platform.logging 1 0 2016-06-15 18:36:03.298
!MESSAGE 2016-06-15 18:36:03,298 INFO  org.talend.components.api.service.internal.osgi.ComponentServiceOsgi  - Registered the component: components#wizards#salesforce(org.talend.components.salesforce.SalesforceConnectionWizardDefinition)

!ENTRY org.talend.platform.logging 1 0 2016-06-15 18:36:03.299
!MESSAGE 2016-06-15 18:36:03,299 INFO  org.talend.components.api.service.internal.osgi.ComponentServiceOsgi  - Registered the component: components#wizards#salesforce.module(org.talend.components.salesforce.SalesforceModuleWizardDefinition)

!ENTRY org.talend.platform.logging 1 0 2016-06-15 18:36:05.939
!MESSAGE 2016-06-15 18:36:05,939 INFO  org.talend.core.repository.model.ProxyRepositoryFactory  - test@talend.com logged on KAAS_CAR_LEAD
....

This open two questions :

  1. Is MacOS is supported ?
  2. Is there some other tricky stuff to do for supporting TOS 6.2 ?

Thanks by advance for your answer.

alexis-gruet-deel commented 8 years ago

no answers ?

Adesin-fr commented 8 years ago

I'm going to give a try on Linux, since I also use 6.2 and need this feature ;)

I'll keep you in touch to tell you my conclusions :)

alexis-gruet-deel commented 8 years ago

This should be great .. Please share !

Adesin-fr commented 7 years ago

Some informations of my testings ;)

I've managed to test it under windows & linux. It works on both, but I had some troubles passing the name of the job(s) to be built : In facts the "regular expression" was a bit complicated to understand, and I had to modify the code to add some loging and get the trick. The problem is that with the current code, we don't have any feedback that the job has been found, or how many jobs are going to be exported. I've also modified the way the plugin selects the job(s) to be exported, I could provide a patch if you want to try it, so now it logs all the jobs founds, and which one are going to be exported !

alexis-gruet-deel commented 7 years ago

Hi @LemarinelNet

Thanks for such feedback ! It seem's the owner of the repository is no longer interested ? May i suggest you to fork this repository and submit a PR ? If you do, i would be happy to give a try of your fixes.

Thanks.

Adesin-fr commented 7 years ago

Here we are :

This is my own version, with logging and 6.2 update : https://github.com/LemarinelNet/talend-job-exporter

Feel free to feedback ;)

alexis-gruet-deel commented 7 years ago

Hi @LemarinelNet,

Just tested and works like a charm. Bravo ! 

[INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
[INFO] Setting property: velocimacro.messages.on => 'false'.
[INFO] Setting property: resource.loader => 'classpath'.
[INFO] Setting property: resource.manager.logwhenfound => 'false'.

Searching for nodes to export matching pattern : /KT_API_TCP/*
   --> Node WS_SOA/** does not match pattern.
   --> Node WS_SOA/** does not match pattern.
   --> Node WS_SOA/** does not match pattern.
   --> Node WS_SOA/** does not match pattern.
   --> Node WS_SOA/** does not match pattern.
   --> Node WS_SOA/** does not match pattern.
  ++> Added node KT_API_TCP

P.S : ( Added /\ for privacy )

Thanks for contribution, this is great for Bamboo, Jenkins or other CI-tools...

Adesin-fr commented 7 years ago

I just have a problem with this plugin : On a server with no graphic environment (no X server running, in example, a Linux Server...), Talend doesn't want to start, complaining about missing GTK libraries... So, I think I'll have to use jenkins on a WINDOWS server :(

alexis-gruet-deel commented 7 years ago

I'm not sure, but if i remember, the -nosplash should avoid the usage of UI/XWindow. Did you tried this switch ?

Adesin-fr commented 7 years ago

Yes I did... on my laptop in linux (with desktop environment, in a console), it doesn't show any GUI, but on a server, it seems that it tries to init some graphical components...

alejobs commented 7 years ago

Hi,

I am trying to run this plugin on my Talend 6.2.1. I have updated pom.xml and plugin.xml with the corresponding versions. However, when I execute the plugin (command line as explained in Readme.md) I can't see what's happening. I am running it on Windows 10. Is there any way of reading stdout?

Thanks!

rbertucat commented 7 years ago

Try running TOS_DI-win-x86_64.exe -vm pathToJDK\jre\bin\java.exe -consoleLog -nosplash ...

ssky1 commented 7 years ago

Hi LemarinelNet,

I am working with TOS 6.2.1 and I am trying to use your version, with logging and 6.2 update : https://github.com/LemarinelNet/talend-job-exporter

The problem is that org.talend.repository_6.2.0.20160510_1709.jar is not the same in the 6.2.1 version wich include org.talend.repository_6.2.1.20160704_1411.jar instead.

When I import your project, I am having compilation errors because
org.talend.repository.ui.wizards don't exist in org.talend.repository_6.2.1.20160704_1411.jar anymore !!!!

So I am wondering why org.talend.repository.ui.wizards got deleted in the TOS 6.21 org.talend.repository jar ?!!!

Is there any work around I can do to make it work in TOS 6.2.1 ?

Thanks a lot

Adesin-fr commented 7 years ago

Hi,

Sorry, I can't help you much on this : I stopped using this plugin, since I hasn't been able to import automaticallly a project in a workspace from command line. My goal was to setup a continuous integration environment (with jenkins).

Instead, my boss decided to buy Talend subscription version :)

Regards

alejobs commented 7 years ago

Hi @ssky1, @LemarinelNet,

There is a way of importing the project into workspace with this same plugin. Take a look at Workspace class. I also achieved to use it with Talend 6.2.1 without any problem. Let me push my changes in a few days.

ssky1 commented 7 years ago

Hi alejobs,

"I also achieved to use it with Talend 6.2.1 without any problem" , did you used org.talend.repository_6.2.0.20160510_1709.jar even with TOS 6.2.1 ( because org.talend.repository_6.2.0.20160510_1709.jar is not available within TOS 6.2.1) ?

I am intrested to see your changes

thanks in advance

Ssky

alejobs commented 7 years ago

@ssky1,

Please look at attached screenshots. Let me know if this works for you. pom_changes plugin_xml_changes

alejobs commented 7 years ago

Hi @ssky1 ,

I looked into my Talend directory, plugins subdirectory, and yes, I found that jar.

ssky1 commented 7 years ago

Hi @alejobs , @LemarinelNet

Finally It worked for me, I could generate the plugin jar and used it succefully to build a talend job on windows.

But When I run the plugin on LINUX (centos) , I am getting this error

com.bsb.tools.talend.export.ProjectImportException: Cannot login on project [XX].

Cannot open the project. Wrong setup of migration task : org.talend.designer.core.generic.model.migration.Salesforce620Migration.

although I am using the same TOS version in both UNIX and Windows and I am testing the same projet

I appreciate your help.

stephend330 commented 7 years ago

[UPDATE] Managed to get this working after updating the Nexus url for talend:

artifacts-oss.talend.com

Hi All,

is this still active? as trying to build now just errors on connections:

[ERROR] Failed to execute goal on project com.bsb.tools.talend: Could not resolve dependencies for project com.bsb.tools.talend:com.bsb.tools.talend:jar:1.1.0-SNAPSHOT: Failed to collect dependencies at org.talend.studio:org.talend.core.runtime:jar:6.2.1: Failed to read artifact descriptor for org.talend.studio:org.talend.core.runtime:jar:6.2.1: Could not transfer artifact org.talend.studio:org.talend.core.runtime:pom:6.2.1 from/to talend_open (http://newbuild.talend.com:8081/nexus/content/repositories/TalendOpenSourceRelease): Connect to newbuild.talend.com:8081 [newbuild.talend.com/68.71.251.200] failed: Connection refused (Connection refused) -> [Help 1]

Any Ideas?

ssky1 commented 7 years ago

Hi,

Why you still want to connect to newbuild.talend.com:8081 (old repo) You have to replace it with artifacts-oss.talend.com, everywhere, in your pom.xml

chmelarj commented 7 years ago

Hi, anyone use this plug-in? I am trying to bring it to live on TOS 6.4.1 but have some issues.

Thx for response.

stephend330 commented 6 years ago

Hi Chmelarj,

I have this working for TOS 6.3.1 so not sure on 6.4.1

What issues are you facing?

Thanks

chmelarj commented 6 years ago

Thx for reply!

I changed the TOS version in pom to 6.4.1 as describe above, then successfully built the plugin and copy the jar files to plugins directory as describe in readme.

But when i run the command for export, in metadata/.log I found error: 1503058108197.log

Then I delete macosx from plugins directory, but got another error: .log

Any ideas?

stephend330 commented 6 years ago

not sure, are you able to post the full command being used?

below is a working example command i have for a centos machine

/opt/buildtools/TOS_DI-20161216_1026-V6.3.1/TOS_DI-linux-gtk-x86_64 -nosplash -data /opt/buildtools/TOS_DI-20161216_1026-V6.3.1/workspace -application com.bsb.tools.talend.export.JobExportApplication -targetFile /home/buildtools/F998_J001_Run_SQS_Requested_Report.zip -projectName talendsource -jobsToExport "DATITUDE/F998_Web_Portal/F998_J001_Run_SQS_Requested_Report*" -clean

chmelarj commented 6 years ago

I wrote down the build process step by step, maybe I am doing something wrong. Please take a look: process.txt The error log is here: 1507120815956.log

stephend330 commented 6 years ago

not sure if you should be copying everthing from target to the plugins directory. i only copied the jar:

bsb-talend-codegen-1.1.0-SNAPSHOT.jar

Thanks

chmelarj commented 6 years ago

Hi, unfortunately did not help, with the same error.

Could you please give me your instructions. I am trying to use it on CentOS, Talend version 6.3.1, how you wrote above, wonder if I am even capable to run this successfully.

Thank you very much.

stephend330 commented 6 years ago

Hi Chmelarj,

Ill do my best to explain however my centos box is a headless server in aws for which i use jenkins to create an X11 Session for talend to display on and then trigger the build using the supplied commands

  1. download talend to centos box
  2. installed jenkins and x11
  3. built jenkins job to create x11 session so talend can start
  4. trigger build using the command supplied

after looking at your error i'm a little confused:

Framework arguments: -application com.bsb.tools.talend.export.JobExportApplication -targetFile C:\Users...\test.zip -projectName TEST_BUID -jobsToExport test Command-line arguments: -os win32 -ws win32 -arch x86_64 -data C:\TOS_DI\TOS_DI-Win32-20170623_1246-V6.4.1\workspace -application com.bsb.tools.talend.export.JobExportApplication -targetFile C:\Users...\test.zip -projectName TEST_BUILD -jobsToExport test

!ENTRY org.eclipse.jdt.launching.macosx 4 0 2017-08-18 14:08:31.119 !MESSAGE FrameworkEvent ERROR !STACK 0 org.osgi.framework.BundleException: Could not resolve module: org.eclipse.jdt.launching.macosx [327]

the error looks to have come from a windows system, but you have stated that you are using centos

Could you post you full command that you are running to export the jobs?

Thanks

ssky1 commented 6 years ago

Hi stephend330,

I have generated the plugin for TOS 6.3.1

but when I execute the command, no error and the build process stops at this message

!MESSAGE 2018-02-02 17:43:28,698 INFO org.talend.core.repository.model.ProxyRepositoryFactory - test@talend.com connected to TEST

My command line is the folowing

C:\outils\TOS_DI-20161216_1026-V6.3.1\TOS_DI-win-x86_64.exe -consoleLog -nosplash -data "C:\Users\toto\Desktop\pluginTalend\workspace" -application com.bsb.tools.talend.export.JobExportApplication -targetFile "C:\Users\toto\Desktop\pluginTalend\export.zip" -projectName TEST -jobsToExport "PROJECTTEST/OR/[0-9a-zA-Z]*" -clean

I notice that PROJECTTEST/OR/[0-9a-zA-Z]* is under C:\Users\toto\Desktop\pluginTalend\workspace\TEST \process

did you modified the plugin's code source to make it work with the TOS 6.3.1 ?

Any Help ?

thanks