Open leandor opened 7 years ago
Interesting point about the Buffer protocol explained here and how to implement it with pybind
Perhaps is something to keep in mind, as a buffer not necessarily means an array of bytes. It could be whatever big chunk of similar sized elements (as pybind example demonstrates), so it could prove useful if there's a similar pattern on some of the record types that require access to portions of it.
Yes - in fact I just started investigating the memoryview
API (replaces buffer in python 3, backported in 2.7). I do so cause I strongly suspect the python patcher can go much faster using this api. As in much.
Nice... We'll need to identify how to proper make use of it and where it's best applied.
My gut feeling is that it's something reserved for big chunks of data that hold the same repeating pattern of elements, so candidates could be PATH GRID / LANDSCAPE records in Oblivion, probably/maybe FACEGEN data (as a chunk of bytes, if needed.) Probably Skyrim/Fallouts have some similar usages.
Could also be something that the client requests for any container that has a non-polymorphic item type, which guarantees same 'record size' for each element, or at least, for which you can produce a representation that has the same size for each element.
For what I gather, any class can have a buffer handler and return an object that has the proper buffer interface.
More docs for my own reference for later: http://stackoverflow.com/questions/3422685/what-is-python-buffer-type-for https://docs.python.org/2/c-api/buffer.html https://docs.python.org/3/library/stdtypes.html#memoryview
I'm starting to see a bit of the road forward for this, so I wanted to write up my ideas.
Why I think this is a good way to move forward?
My idea is to avoid touching internal CBash code as much as I can, to avoid introducing new bugs/regressions while there are no tests cases that protect the code, so steps 1), 2) and 3) can be made with probably almost no alteration of the inner workings of the existing deep 'core' stuff, like record handling, mod file parsing, etc. And I hope that by the time cases 4) and 5) enter the picture, there will be already more test coverage and protection against more common regressions.
Of course, I'm open to suggestions, alterations, rebuttals and comments :)
Well - it just occured to me that I have a binary commit ready on my branch with 7z version 16.02 - if we are to look into the python bindings for 7z dll then maybe it's the time to do it to avoid a double binary commit. On the other hand I do want 16.02 in the beta due the vulnerability discussed here https://github.com/wrye-bash/wrye-bash/issues/313 so I might as well go ahead and commit and we revisit this at our ease. The beta has some work ahead - including loot, merging my wip branch which needs more work etc
It depends on the time frame you'd want to wait. I mentioned the 7zip binding as a long term project, but if you need to focus on it, I could switch from CBash and start looking into it.
I don't even know yet if the provided library has all the functionality needed for Bash, so it'd all depend on that.
It's something that would take me between 2 weeks and 1 month to figure out and implement, I think... and assuming that I can keep doing stuff while I'm on work hours (while my work is not too demanding.)
Even then, I'd like to think a proper solution to the dependence problem... as in, dependence in self produced binaries, not on 3rd party stuff.
How other projects handle this?
We should look for projects that have many modules that compile independently here in GitHub, and see how they handle integration, IMO.
In any case, I'd think that having a whole Git repos for handling releases (or, probably, one for each project that produces binaries would be the ideal, maybe) is not something too problematic (at least in the mean time, until a proper solution can be devised.)
AFAIK, the whole "do not commit binaries to GIT thing" is when you have source and binaries intermingled, but if you dedicate a whole repos for releases, I don't see the problem... Git handles binaries very well, and even if they become too big to handle, you can always delete the whole repos and regenerate it, since you aren't really interested in the history, right? As long as you keep one branch of binaries for the stable, and one branch for dev (and maybe, an extra branch for experimental releases, occasionally) it just shouldn't become too big too handle, IMHO.
((As an example, in my work, we have a separate SVN repos just for the released binaries, that has a mirrored structure w.r.t the source repos, and the actual source working copies bring the release directory from that release repos using SVN externals. The build bot just compiles the source and then auto-commits the produced binaries that end up in the release dir, which commits to the binaries repos.))
And I suppose that another solution could be to create 'proper' Python packages using setuptools/pip and host them on PyPI (assuming PyPI even host the binaries themselves? I know they host source tarballs in some cases...), but I don't know how much work would that option entail.
I've been toying with a continuous integration system on a personal fork of this repo (just in case I messed up) and it seems to work good.
I found AppVeyor and decided to test it since it's free for open source projects (with limitations) and supports the environment we need.
There's also travis-ci.org, which is better IMO, but it doesn't support Windows builds (yet).
I'm gonna play with Travis maybe tomorrow, to see if I can make CBash compile in a Linux environment.
See here for the build that passed.
I've already read up a bit on how to generate releases automatically and how to handle dependences.
Also, here's the docs on Releases: https://help.github.com/articles/creating-releases/
Next thing I'm gonna do is to try to optimize the build time a bit, because compiling everything takes 15 mins, and I'm already dreading having to change the code due to the compilation times.
So, first step is to try the compiled headers stuff I found around, and the second step is to split up CBash project in several static libraries that get linked together at the end.
After that, I'll be starting code the step 1) on the list above.
Welll of note that apparently (https://github.com/wrye-bash/wrye-bash/pull/329 ) we have some serious problems packing the pyd file with the standalone - I had similar problems making scandir work (a python module that uses a pyd and depends on the msvc)
Hmm... that worries me a bit, as soon as I have a working binding for CBash, I'd take some time to do some experimentation too with the standalone version.
Yes I'll turn the eye on this too - anyway I want scandir to work too, speeds up quite a bit filesystem listing.
The pyd has no problems being packed, it's the (LOOT API) DLL that it calls that I haven't been able to pack.
Hey Wrinkly,
Maybe you can just solve it by compiling against a static version of that lib when building the pyd?
That would mean having two different builds, if you need the DLL output, as I don't think CMake allows generating both the shared and static versions of a target from the same build process.
If you link statically then you need to be very careful about memory allocation, because your library will allocate memory in its own heap, so you can't have objects get freed externally (they'll try to free memory on a heap they don't exist on).
If you're providing a C API this isn't an issue because the memory management is all explicit anyway, but I found (at least with pybind11) that leads to pointer hell, primarily re: output argument pointers, which just don't map to Python (which is why I guess ctypes constructs object wrappers for pointers, but which is probably a factor in its large overhead).
EDIT: I'm talking about linking statically to the MSVC runtime here, if you mean linking to the library itself and not the runtime, then you pull that runtime linking into the pyd, which would probably just move the issue from the DLL into the pyd, leaving importing it totally broken.
Yes, I meant just for your (intermediate, at least from the pyd
standpoint) DLL library.
Agreed, even if you link to it statically, the C/C++ runtimes (and python runtime too) would always have to be linked dynamically, as you said.
Steps and tasks
Step 1
Step 2
Step 3
Step 4
Step 5