Inability to move binary data between Lua strings and Java

[GoogleCodeExporter]

When sending data between Lua and C using the Lua C API, there are two sets of 
methods for moving character data, the "string" methods and the "lstring" 
methods.  Below is the list of lua_ and luaL_ methods that deal with "strings" 
and provide both versions:

lua_pushstring / lua_pushlstring
lua_tostring / lua_tolstring

luaL_addstring / luaL_addlstring
luaL_checkstring / luaL_checklstring
luaL_optstring / luaL_optlstring

In all cases, the "string" value is referenced on the C side as a "const char 
*".  On the Lua side, a "string" is defined and implemented as follows: 
"Strings have the usual meaning: a sequence of characters. Lua is eight-bit 
clean and so strings may contain characters with any numeric value, including 
embedded zeros. That means that you can store any binary data into a string."

So Lua strings exchanged with C code may be either "text" or "binary", with the 
determination being made by contract between the Lua and C code.  The fact that 
the Lua strings may be text or binary is precisely why the lstring functions 
exist.

For example, lets say that you implement an HTTP request API in C as a Lua 
extension.  When Lua calls that API to upload a file, it needs to pass a binary 
request body, say by loading an image file into a Lua string and passing that 
string to the C API.  Since the C API is expecting binary content in the Lua 
string, it uses the lstring functions, probably lua_tolstring, to access the 
data, and processes the resulting char array as binary bytes.  On an HTTP image 
download, the C API would to exactly the reverse, pushing the binary content 
onto the Lua stack using push_lstring, and the Lua code would then receive a 
Lua string with the binary content.

Unfortunately, the LuaState string APIs supplied by jnlua do not implement the 
lstring methods (though they do call them internally in the JNI bridge).  The 
jnlua string support methods are as follows:  

String checkString( int index )
void pushString( String s )
String toString( int index )

All of those methods move data between Lua strings and Java as UTF8 encoded 
text (so they assume that the Lua string is or should be UTF8 encoded text, and 
encode/decode such that the Java side representation is a Java String).  As far 
as I can tell, there is no way to move binary data between a Lua string and 
Java using the current implementation of jnlua (v1.0.2).

The solution to the above issue would be to implement versions of the three 
string functions currently supported by jnlua, but that work with a Java char 
array or byte array (one or the other), possibly something like:

byte[] checkStringAsBytes( int index )
void pushStringAsBytes( byte[] bytes )
byte[] toStringAsBytes( in index )

These obviously would just use the Lua C lstring functions without doing any 
encoding/decoding of the data on the C/Java side.

As it is, the inability of jnlua to move binary data between Lua string and 
Java is a showstopper. 

Original issue reported on code.google.com by bob.dick...@gmail.com on 2 Oct 2012 at 11:04

[GoogleCodeExporter]

This makes sense. I would probably go for more compact mathod names 
pushByteArray() and toByteArray(). Also, some extensions to the default 
converter are meaningful, allowing byte[] on the Java side to drive the 
corresponding conversions.

Original comment by an...@naef.com on 3 Oct 2012 at 12:22

[GoogleCodeExporter]

I suggested having "String" in the API names just to make it clear that the 
interaction on the Lua side was with a Lua String, but I'd be fine with your 
suggested names.

I'm working with Corona Labs, which embeds jnlua.  It appears that they are 
using v0.9 (per LuaState.VERSION).  They are aware of this issue and I think 
they're willing to upgrade to the most current release in order to get a 
solution to this issue.

Thanks.

Bob

Original comment by bob.dick...@gmail.com on 3 Oct 2012 at 6:28

[GoogleCodeExporter]

Changesets adding pushByteArray, toByteArray and checkByteArray as well as 
adapting the default converter have been committed to the repository for 
testing. The pre-built binaries for Windows have also been updated. Unless 
there is negative feedback, this will be released as JNLua 0.9.5 and 1.0.3 
respectively in a few days.

Original comment by an...@naef.com on 5 Oct 2012 at 11:13

[GoogleCodeExporter]

I did not build and test this, but I did review the new APIs and did a code 
review of the implementation, and it all looks good to me.

Original comment by bob.dick...@gmail.com on 7 Oct 2012 at 11:31

[GoogleCodeExporter]

Released as JNLua 1.0.3 and 0.9.5 respectively.

Original comment by an...@naef.com on 9 Oct 2012 at 11:12

[GoogleCodeExporter]

Original comment by an...@naef.com on 9 Oct 2012 at 11:15

• Changed state: Done