Add C source code to Python install and add support scripts

[rowleya]

If the C source code were included in the sPyNNaker python package, it would automatically be placed on the user file system when a user installs sPyNNaker. In addition:

• add a get_neural_modelling_dirs.py script, which outputs the path to the neural_modelling directory to stdout
• add c_setup.sh scripts that exports the result of get_neural_modelling_dirs.py as NEURAL_MODELLING_DIRS

The user now simply has to call c_setup.sh from their .bashrc file. There may also be better ways of achieving this.

[alan-stokes]

not getting what this achieves in general. whats the aim?

[rowleya]

The aim is mainly that the user no longer has to download the Python and C code just to get at the C code when making their own models. Because this then puts the source code in a random location on their hard disk, we can use Python to extract that location and make it easier overall for them to set things up.

[alan-stokes]

still missing something here. developer version c code in module locations, and requires those environmental variables to be set if you want to not use automatic make.

the pip version has no c source code, but the binaries. what am i missing?

[rowleya]

It would have source code if we made the change suggested here... then the user will have the C source code automatically.

Using automatic make would still work - possibly even easier since you could set the environment variables in the automatic make script, so they don't even need to be set at the start.

[alan-stokes]

so, lets see if I'm getting this correctly. you want to put the c source code into the pip releases, so that end users whom use pip can do extra models. Even though the definition of the pip release is pure pynn, no messing with gubbins, and that's what the developer releases are for?

the pip users would then need to install the c compiler, and now you're compiling in some .local/lib location, which likely has odd permissions. Whereas the developer install resides in a user directory with their permissions for compiling.

if thats the logic, i dont like it. if its something else, im still lost.

[rowleya]

Sounds like you are a little lost. It may be easier to split users into 3 categories:

1. Regular PyNN user. Just runs PyNN models that we already have.
2. Extension PyNN user. Uses Pynn but wants to add new models and / or plasticity rules.
3. Developer. Wants to change underlying code.

Currently we support 1 and 3 well but Users in group 2 have to install PyNN and then download the source code of sPyNNaker and SpiNNFrontEndCommon (both C and Python) so that they can build binaries. If we included the spynnaker neural_modelling folder in the spynnaker install, they would avoid downloading incorrect versions. This could even include the spin1 api, sark and fec libs required to make it extra easy. They then just install a C compiler and off they go.

Note we would still also include the aplx files in the release as we do now for user group 1.

[alan-stokes]

ok (so wasn't lost), but now your having to give group 2 all of the developer stuff without actually going through the developer install. Its more work, more documentation, and really. Given that you then asking option 2 to install c compiler, adding environmental variables, and everything else.

This would just boil down to developer version via pip instead of via github, and by doing it in pip, you've got stuff in .local or where-ever python installs its code. So now your getting them to compile in odd locations, with odd permissions. This is just adding workload and potential for annoying issues. keeping group 2 in the same frame as group 3 seems the safer approach

[rowleya]

OK, so to make it clear, here is what we give each group to install:

1. Install Python then pip install sPyNNaker7/8
2. Same as above but then download the C compiler, spinnaker_tools (must match release version), spinn_common (must match release version), SpiNNFrontEndCommon source code (must match release version), and sPyNNaker source code (must match release version), then set up SPINN_DIRS and NEURAL_MODELLING_DIRS.
3. Download and install the latest git master, and C compiler.

The issue with lumping group 2 into group 3 is that they aren't group 3 - so they now have installed PyNN release and now have to remove that to install the developer version. This might be buggy since it is just the github master.

If we change it to what I suggest above, they can simply do the same as group 1, then install a C compiler. We could even provide them with an automatic make script as you suggest which would do the setting up of the appropriate directories (spinnaker_tools would become a path within the spynnaker install as well, though it wouldn't include everything of course, just headers and libraries).

An alternative here is to have an additional pip install that has no python code to speak of. This would simply contain the source code of sPyNNaker/neural_modelling and the appropriate build libraries and environment.

[andrewgait]

I agree with Rowley here - option 2 as he sets it out would be a much easier way to support a user like Harry for a start...

[alan-stokes]

but they don't need to go to master for developer install (i concede that's what we point them at currently). But the tools are tagged at release points. Just option 2 people are often already option 2 to begin with, so we should instead of going into group 1 then having to go to group 3, should have jumped to group 3 already. we'd need to define what these groups are up front at install time.

The main issue i have with this is 2 fold (excluding harry which is another case entirely).

1. you've now given group 1 c code, which is pointless to their requirements (not too big of a issue really).
2. you've now given group 2 a potential pile of permission issues in compiling, as they will have installed it in places like .local/lib etc, which even on a normal linux machine, has locked down permissions. I think you'd need to be sudo to do the compile. And that's going to add issues to us, not reduce them. Telling them up front to install the correct stuff in the first place is a better approach, and creating a developer doc for each release is no big deal.

[andrewgait]

An alternative here is to have an additional pip install that has no python code to speak of. This would simply contain the source code of sPyNNaker/neural_modelling and the appropriate build libraries and environment.

Looking back over this issue, this is probably the best solution and doesn't require much work from our end. It's not that different from what we would currently ask users doing the new neuron models lab to do...

[dkfellows]

I would love to have the C in a separate repo anyway, both for sPyNNaker and FEC. The current approach is a bit too "mulch together" for my taste.

Low priority for fixing though. Very low.

[Christian-B]

Agreed on low priority.

Putting all the code together and in the same structure style could also simplify our make files and possibly allow https://github.com/SpiNNakerManchester/SpiNNFrontEndCommon/issues/414

[alan-stokes]

i agree that having a set of repos for the c codes could be usful.... we'd need a python module in there, so we could get the exuectable finder to import it to find the path. but could be done. the issue is it would be a very costly merge hell for any open prs. lol