[Paper+1] Garbage Collection in JyNI – How to bridge Mark/Sweep and Reference Counting GC Stefan Richthofer

[Stewori]

This paper is still somewhat incomplete. I already submitted this version, because no exact submission deadline time for September the 15th was defined. Main things that remain to be done is writing the introduction and a chapter about weak reference support. I am going to finish these tasks today (i.e. on the 15th), so this is actually in time if I can assume that the 15th is entirely within the deadline.

[Stewori]

Also note that my fork of euroscipy_proceedings still contained my proceedings paper from 2013. I pulled all updates from euroscipy_proceedings into my fork and replaced the 2013-paper with the 2015 one. However, now I noticed that my pull-request also lists actions done to the 2013 paper. For current EuroSciPy just ignore all but the last item "Updated paper to EuroSciPy 2015".

For some reason my paper is still rendered with the page header "PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013)". I don't understand how this comes, since I recently pulled all updates from euroscipy_proceedings into my fork. Any advice how to fix this?

[pdebuyl]

Hi Stefan,

Your master branch is in conflict with the current repository, see https://github.com/Stewori/euroscipy_proceedings/blob/master/euroscipy_proc.json#L8 where the 2013 edition is still referenced.

[Stewori]

Alright. Confusing that github did not report this when I merged the 2015 variant into my fork. Actually I wanted to create a different fork for the 2015 proceedings, but github does not seem to allow multiple forks of the same project within one account. However - thanks for the hint!

[Stewori]

I tried to solve the conflict with the 2013 variant and it appears, there actually is no conflict; https://github.com/euroscipy/euroscipy_proceedings/blob/master/euroscipy_proc.json is already outdated in the base-repository. Or is there another EuroSciPy proceedings repository than euroscipy/euroscipy_proceedings that I should have forked? For now I'll fix euroscipy_proc.json in my fork manually up to my knowledge and include it in the next pull request.

[pdebuyl]

Indeed, the current master has an outdated file euroscipy_proc.json. My local version was not in sync. I'll let @NelleV manage this then.

[stefanv]

Comments from a cursory first glance:

• Please do a grammar check on the whole paper.
• The abstract is an important part of the paper. You can improve it by providing a better idea of what JyNI is, and by clarifying the purpose of this specific paper.
• Add references to claims such as claims that Python considers switching to mark-and-sweep.
• Cite the other projects mentioned, such as CFFI.
• The logo figure is superfluous.

[Stewori]

• I updated the abstract and added references for many mentioned projects. I think it is not required to reference extremely popular projects like NumPy, PyPy, IronPython or Python itself, is it?
• I removed the logo figure like you suggested, but actually would like to have the logo on the paper in some way, so if there is a proper way to do so, please let me know. If logos are not wanted in principle I will of course accept that.
• The claim that Python considers switching to mark-and-sweep is from Larry Hasting's talk at 2015 Python Language Summit and was already referenced (https://lwn.net/Articles/640179). I think I made carefully clear that these are rather vague considerations, still underlining the relevance of the topic. I did not mention the reference in the abstract originally, because I somehow thought that abstracts should not contain references. I did not yet check for other claims that might need references (but will do so soon); if you have some specific in mind, please feel free to point me to it.
• I checked the paper for grammar issues, but it mostly appears okay to me. Still I improved various minor things, e.g. I am having a hard time to decide whether to use a "-" to connect words or not. I queried several terms on the internet to look up common style but still didn't really get the rule (e.g. it appers to be "problem-specific", but on the other hand "Python specific".) Also apologize if I still overlooked spelling issues. Auto-correction is messy, because the text contains so much computer-science specific vocab. However, I will give it another look.

[stefanv]

The logo can be incorporated into a diagram, if such a diagram exists. It's just a bit strange to have it hanging around without a caption.

I wouldn't reference NumPy, but I would reference PyPy, CFFI, and other projects that do similar things.

Vague claims can still be cited--you're not stating that Larry promised anything. The sentence outlining the claim is one of those for which the grammar bothered me. "Vague claim", as you use here, is very acceptable.

Hyphenation preferences vary quite a bit; I don't think it's a big deal. The correct use of an em-dash (rather than an en-dash) to connect sentences would be good.

[arigo]

Sorry for the delay. As far as I can tell, this is similar to ideas we're playing with for cpyext, which is PyPy's CPython C extension module compatibility layer. This paper goes into more depth than we do, e.g. supporting reference cycles, but on the other hand cpyext is more practically useful right now and we're seeing issues that occur in practice. There are a lot of them that the authors seem to ignore so far, and numpy is particularly troublesome, so I'd be careful when presenting the work here as getting most of the way towards numpy support. The approach is still interesting and well-presented, though.

[arigo]

For reference, CFFI is very different: it is a layer to access generic C code. This is instead about C code that is written specifically for one VM, and we're running a different VM for the same high-level language.

[stefanv]

Thanks very much for taking time to review the paper, @arigo. @Stewori If you can update the paper with this latest feedback, then we're good to go from my point of view.

[Stewori]

Also sorry for the delay; I am very busy right now and hope to find time for the final editing soon. My intention was to present numpy as the driving motivation for JyNI, which is simply plain truth. I'll check, where I might have stated this in a too optimistic manner. So far I could hardly work with numpy itself, because essential dependencies were not yet working (e.g. ctypes is currently gaining support). I am highly interested to learn in detail about the practical issues you mention (@arigo maybe we could get in contact about this topic independently from this thread). It's not that I am ignoring them; such issues were - so far - just out of scope due to higher priority on improving the general concept - supporting gc, weak references etc. That said, keep in mind that JyNI is at least internally very different from cpyext, mainly indicated by the fact that it is (in principle, modulo uncompleted API) binary compatible to extensions, while cpyext requires recompilation with different headers. Given the tons of macros Python is using, it is literally impossible to tell how issues from cpyext would transfer to the JyNI concept. No doubt - there will be issues with numpy and other extensions - gonna face them one by one.

Regarding CFFI, it was never my intention to compare it with JyNI; I rather target CFFI as one of the key extensions to support. From CPython's point of view CFFI (i.e. the CPython version) is an ordinary C-extension, isn't it (one that accesses gcc though)?

[arigo]

Hi Stefan,

On Thu, Dec 3, 2015 at 2:05 AM, Stefan Richthofer notifications@github.com wrote:

Regarding CFFI, it was never my intention to compare it with JyNI; I rather target CFFI as one of the key extensions to support. From CPython's point of view CFFI (i.e. the CPython version) is an ordinary C-extension, isn't it (one that accesses gcc though)?

Ah, sorry, I didn't get that. Then I'm pretty sure that a better idea would be to natively port CFFI to Jython. It is not a lot of work; in fact the design of CFFI is such that it is meant to be ported in this way, as is done with PyPy. It doesn't expose anything to the user that only makes sense on CPython. Moreover the interesting parts of such a port would be to tweak it in order to improve performance inside Jython; that's how CFFI is much faster on PyPy than on CPython, after the JIT kicks in.

Of course, it is more interesting to think about general solutions, but in this specific case I believe we can achieve much, much better results with a native rewrite.

A bientôt,

Armin.

[Stewori]

Then I'm pretty sure that a better idea would be to natively port CFFI to Jython

I agree, but this would be a distinct project and I don't have capacity to do this too. AfaIk https://github.com/jythontools/jiffy (c.f. http://bugs.jython.org/issue2148) is exactly about that, but I don't know about its current state as the page is currently just a stub (which is why I didn't mention it in the paper).

However the JyNI variant will provide an alternative that is potentially available earlier; "works unperformant" is still way better than "unsupported".

Another sorry for the delay; I am currently at ICMLA and was busy with preparation before. As soon as I am back from this conference I'll follow up on this.

[Stewori]

I applied the changes suggested by @arigo, updated the ctypes-section, corrected some typos, added references for PyPy and IronPython, added a footnote about CFFI and applied various other improvements.