Python binding improvements - mark 2

[ikelos]

Hiya, I just tried building the python bindings for linux and found pull request #14. I've tried to update it to the support the latest version, but I have to admit my C code isn't all that great.

It compiles just fine, but LcCreate fails and I've no idea how to add the structure needed to LcCreateEx to see what's actually causing the problem. On the same system I can run leechcore with the same device parameter and it reads just fine.

I'm happy to help test and develop improvements, but my C skills are pretty basic. Anything that can help drive this forward a bit would be appreciated. It feels like so much work has already been done, there's just a little bit extra needed to get linux supported too... 5:)

[ikelos]

I agree that packaging for pypi containing binary components is likely difficult. I guess look at things like distorm/capstone for how to do that?

The README doesn't really explain why you don't accept contributions, but that's fine. The changes aren't a significant work and I fully trust you could and possibly will make them on your own, but if you don't have time to work on python support this may help motivate others to try it out and develop it. If you get a CLA in place I'd probably sign it, I'm not really in it for profit I just want a cool tool that everyone can use at the end of it. 5;)

I'd really like to see example code with LeechCorePyc so that the compiled library can be tested with known working code and possibly also to give the API a feel and see if it can be made easier (you might want to stick with the LeechCore API as closely as possible, but if you'd done the work of making a python API it might be worth deviating in order to make it more pythonic?). I don't have any figures to back up the claim, but I suspect that a strong interpreted interface like python will get people playing with the technology more than having to wrestle with C to then play with it.

Thanks for the pointers, I'll check out the optional driver. The fact that it works on the system installed pcileech suggest that's not the problem, but it might be a linking issue or something, given the python module seems to be a complete rebuild of the library. I was looking at a CFFI interface but all the defines for Windows/Linux made that a little tricky so I haven't gotten too far with it (particularly once I found leechcorepyc).

Errr, I don't think build from source on linux is necessarily a must, but offering people the possibility will likely go down well (I use a source-based distribution, so I much appreciate having the option). It will also mean that the build process will get stress tested and you'll end up with quite a robust system if you have the time and energy to fix the bugs that'll get filed. 5;)

Ah, ok, I thought it might be for general errors like "can't find library" or "USB device not plugged in" or similar. It would be useful to expose such troubleshooting responses from the call, but I understand if that's tricky. I haven't gone anywhere near the agent yet, so I don't have a great deal to comment on it.

Getting it to build (and work) on linux with instructions (so pointers about where the FPGA libraries need to be etc) would be awesome! Packaging it as a binary might be nice to make it slightly more convenient but it's certainly not critical to me personally.

In short, any help you can provide would be appreciated and if you can help get past the hurdles so I can start building on it, then I can make more use of it and contribute/suggest things that might make stuff easier. I'm also happy to wait for a CLA before expecting merges... 5;) Thanks Ulf!

[ikelos]

Not sure if this is the right place for discussion, happy to move it elsewhere if there's somewhere better.

I managed to get it going and found the following:

• Even changing the python library to build against the library rather than compile a copy into the module, it still always looks in the directory that the executable is run from to find plugins. This means /usr/bin because it's python that's running and python lives at /usr/bin/python.
• LEECHCOREPYC_Open works fine, but rather than returning you an object containing the handle (that could then have methods called on it, like read, etc), it opens a global handle, meaning you can only have one open at a time, and calls like LEECHCOREPYC_Read only ever act on the one global open handle.
• LEECHCOREPYC_Read(offset, length) didn't seem to work (I tried multiple offsets and multiple lengths, I only ever got an empty bytes object back)?
• LEECHCOREPYC_ReadScatter([list of offsets]) worked fine, and returned me the pages that had been requested in the list. This is also what leads me to believe that LEECHCOREPYC_Read isn't working, because the same offset put in ReadScatter worked fine.

So things that would make this much better/easier for linux users to use would be:

• Move the library to libleechcore-x.y.z.so following the standard library numbering scheme (https://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html). It's very similar to semantic versioning if that helps? It needs to start with lib so that all the linkers and so on can find it (-lleechcore in gcc wil look for libleechcore when searching for the library).
• Load the plugins from the library's/a fixed location, rather than the executable using the library.

So things that would make this much better/easier for python users to use would be:

• Have LEECHCOREPYC_Open return the handle and pass that in to other LEECHCOREPYC calls.
• Drop the LEECHCOREPYC at the start of the function names, it's a bit cumbersome to put in, particularly since the module is already called leechcorepyc. 5;)
• Refactor the LEECHCOREPYC calls, so instead Open creates an object and the other functions turn into methods on that object, so that you could end up writing something like:

  lc = leechcorepyc.open('fpga://', '')
  data = lc.read(0x1000, 0x2000)
  lc.close()

I know there's a lot there and it's absolutely a niche part of a niche project that probably doesn't consume all your time in the first place, so no problem on if/when it gets done, but I figured I'd let you know what I think would help and then at least it's out there. I can at the moment pull data back using what's there, it's fully usable as is, so thanks for that!!! 5:D If you are interested in making any of the changes, or get a CLA that you're happy with then I'm happy to help test, etc where I can... 5:)

[ufrisk]

I'm closing this for now. The plan is to release an updated version of this python binding within the next 1-2 months; and possibly on python pip as well as discussed separately.

[ikelos]

Just noticed a couple more things, and this seems like a reasonable place to keep track of them, even though the issue's closed:

• It would be really handy to know how big the accessible memory is, I've done this in the existing implementation by adding: PyDict_SetItemString_DECREF(pyDict, "maxAddr", PyLong_FromLong(cfg.paMax));. If it's being refactored into a python object, then having this accessible in the object would be massively useful.
• The LEECHCOREPYC_Read function currently fails because it uses cbRead which is set to 0 at the start of the call and then never changed. It starts returning results when cbRead is changed to cb (although I haven't chekced if the LcRead call will change cb to say how many bytes were actually read).
• Even after changing that LEECHCOREPYC_Read always reads from offset 0, because the argument doesn't get converted properly. I don't know why, parsing the args tuple looks ok, the format string looks ok, but pa always comes back as 0 (even if it's initialized to something different).