search

pentaho-labs / pentaho-cpython-plugin

This is a PDI plugin that allows execution of Python code.
Apache License 2.0
32 stars 19 forks source link

Works In Spoon. Fails in Pentaho (when executing via scheduled reports server) #31

Closed igorpetrenko3690 closed 3 years ago

igorpetrenko3690 commented 3 years ago

Hello,

Apologize in advance if this is a simple issue that has been previously covered.

Spoon is installed on the same box as the Pentaho server. I have Anaconda3 installed. Spoon is able to find all libraries referenced in the code and executes the Python job fine/successfully.

When executing the job via the Pentaho Scheduled jobs I get an error (see stack trace below) How do I install the "sharepy" module? It is 100% installed within Anaconda as Spoon is able to make use of it and finishes the transformation successfully. Did I misconfigure Pentaho server? How can I add sharepy to Pentaho server? Any help is appreciated.

2020/12/07 09:59:42 - Python SharePoint Sync - Dispatching started for transformation [Python SharePoint Sync] 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Unexpected error 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : org.pentaho.di.core.exception.KettleException: 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - Traceback (most recent call last): File "C:\pentaho-server-ce-9.0.0.0-423\pentaho-server\tomcat\temp\pyServer.py", line 299, in execute_script exec (script, _global_env) File "", line 6, in ModuleNotFoundError: No module named 'sharepy'

2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at org.pentaho.di.trans.steps.cpythonscriptexecutor.CPythonScriptExecutor.executeScript(CPythonScriptExecutor.java:452) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at org.pentaho.di.trans.steps.cpythonscriptexecutor.CPythonScriptExecutor.executeScriptAndProcessResult(CPythonScriptExecutor.java:349) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at org.pentaho.di.trans.steps.cpythonscriptexecutor.CPythonScriptExecutor.processBatch(CPythonScriptExecutor.java:338) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at org.pentaho.di.trans.steps.cpythonscriptexecutor.CPythonScriptExecutor.processRow(CPythonScriptExecutor.java:243) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - at java.lang.Thread.run(Thread.java:748) 2020/12/07 09:59:42 - Python SharePoint Upload (Folder Sync).0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=1) 2020/12/07 09:59:42 - Python SharePoint Sync - Transformation detected one or more steps with errors. 2020/12/07 09:59:42 - Python SharePoint Sync - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Errors detected! 2020/12/07 09:59:42 - Python SharePoint Sync - Transformation is killing the other steps! 2020/12/07 09:59:42 - Python SharePoint Sync - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Errors detected!

Thanks, Igor

eicherjc commented 3 years ago

The CPython step executed by the Pentaho Server should be using whatever Python is on the PATH for the user that the Pentaho Server was started with. Is the user profile that's started the Pentaho Server a different user than what started the instance of Spoon that completed successfully? Additionally, if you're using Python virtual environments at all, that could be contributing to the issue. Pentaho Server will use just whatever is in the PATH and not necessarily use a python virtual environment, unless you activate that environment as a part of the Pentaho Server start script.

igorpetrenko3690 commented 3 years ago

I appreciate a fast response, Chris, and your willingness to assist despite this question not really being the bug I think (more of a config issue on my end).

I have configured Pentaho server to run as a local system account as per the below included screenshot. image

Spoon is launched using my user (who is a local admin user).

I have looked into the activation of the Python Environment, which btw I am using "base". If I understand correctly you are recommending to add a line "conda activate base" somewhere in the start-pentaho.bat file?

It does appear to me that the Pentaho server does not access the correct Python environment and I am still on the hunt for the exact steps that will fix that.

System Path variables: image

Pyhton Environments (base being set to auto_activate_base true): image

Thanks in advance, appreciate any and all help I can get. Igor

igorpetrenko3690 commented 3 years ago

Fixed with a bit more elbow grease :) 100% a misconfiguration on my end and a good lesson learned. Thank you!