A utility to read DLPX_OUTPUT_FILE would help ease Lua to Python transitions

[fdrozdowski]

From Hemant on slack: Hi

Thanks, I’ll check fix tomorrow or on Monday.

I have one more thing to discuss, as per our current Lua implementation we write json output in DLPX_OUTPUT_FILE and this variable is treated as outschema for corresponding Lua script. By this way we can distinguish between standard output of bash script and outputschmema, for an example we have a lua repositoryDiscovery.lua for discovery and using RunBash function we are calling discoverRepository.sh and bash script is writing json content in DLPX_OUTPUT_FILE and apart from this we are also printing few line in bash (using echo or printf).

In Python Plugin we don’t have any way to distinguish between standard output and output schema although we can make few changes in bash (or can read specific json data from stdout of run_bash response) and everything will work fine.

Can we have some method at VSDK end where we can read DLPX_OUTPUT_FILE from run_bash response? I think this will ease the transition/migration of Plugins from Lua to Python with almost no change at bash scripts side.

┆Issue is synchronized with this Jira Story by Unito

[fdrozdowski]

➤ Tom Walsh commented:

Overview

Currently, if a Lua plugin wants to pass information from Bash/Powershell (on the remote host) to Lua (on the engine), that works like this:

1) The plugin's Lua code passes an outputSchema to runBash/runPowershell. This is a JSON schema that defines the expected format of the information. 2) Appdata's runBash/runPowershell code sets up a temporary file on the remote host. 3) The bash/powershell code is run with an environment variable DLPX_OUTPUT_FILE that points to this temporary file. 4) The bash/powershell code writes to this file. It's expected, of course, that the contents of this file end up being a JSON instance that validates against the outputSchema. 5) Appdata's runBash/runPowershell code reads the temporary file, then deletes it. 6) The contents of the temporary file are converted from JSON to Lua, and then returned to the plugin's Lua code.

This ugly design was put in place due to the fact that text processing is extremely limited in Lua code. Granted, it is a gigantic pain in the neck to construct JSON from Bash/Powershell code, but even that is preferable to doing string processing in Lua. So, this is a Lua-specific design.

In the Python world, though, none of this is necessary. The bash/powershell code can simply pass back any text it wants to, because Python has powerful and convenient text processing abilities.

Proposal

This Jira ticket is proposing that we simulate or duplicate this setup with Python plugins.

Pros:

• Would help ease the transition from Lua to Python. No Bash/Powershell code would need to change, and Python code could be basically a direct "translation" from Lua

Cons:

• "Pollutes" our platform API with unnecessary stuff
• Requires AppData to continue to support this Lua-specific design well into the future
• At best, it makes it harder for new plugin writers to understand the API. At worst, it might encourage them to start using this technique, when there are better techniques available.

[fdrozdowski]

➤ Tom Walsh commented:

There are basically two alternatives I can see for Lua-to-Python conversions that require no support from Appdata, apart from the "real fix" of moving string processing entirely to Python and abandoning the practice of creating JSON objects from Bash/Powershell.

Alternative 1: Use STDOUT instead of DLPX_OUTPUT_FILE for JSON

1) Modify Bash/Powershell code as necessary to eliminate anything that is currently writing to stdout (e.g. by redirecting to /dev/null) 2) Modify Bash/Powershell code so that anything currently writing JSON to $DLPX_OUTPUT_FILE will write to stdout instead. 3) In Python, you can use json.loads(result.stdout) to convert the serialized JSON to a Python object.

Alternative 2: Simulate old Lua behavior inside toolkit

1) Write a Bash/Powershell script that creates a temp file on the remote host and prints the filename 2) Write a Bash/Powershell script that prints a file to stdout, then deletes it. 3) Write a Python function that ties this all together... something like this (pseudocode)

def old_style_run_bash(connection, command, variables, outputSchema): // Create a temp file on the remote host temp_filename = libs.run_bash(connection, create_temp_file_script).stdout

// Add the temp file to the list of variables variables['DLPX_OUTPUT_FILE'] = temp_filename

// Call the real bash script libs.run_bash(connection, command, variables)

// Read and delete the temp file result = libs.run_bash(connection, read_and_delete_file_script)

// Convert the JSON instance from the temp file contents to a Python object output_object = json.loads(result.stdout)

// OPTIONAL: Actually validate against the provided schema. jsonschema.validate(instance= output_object, schema=outputSchema)

// Finally return the Python object return output_object4) Instead of calling libs.run_bash(...) from Python code, instead call old_style_run_bash(...)