search

automate-lfs / jhalfs

Tools for running the Linux From Scratch book instructions
http://www.linuxfromscratch.org/alfs/
MIT License
66 stars 14 forks source link

Add a Python entrypoint and packaging config #1

Closed jhuntwork closed 5 years ago

jhuntwork commented 5 years ago

This commit will turn the source into a Python package which can be packaged using setup.py and/or installed via pip.

When installed, a new binary named 'jhalfs' will be placed in the system PATH that serves as a wrapper for the main() method in the jhalfs module. All the previous scripts are moved 'as-is' into the module's path and will be packaged with the module.

At run-time 'jhalfs' creates a $HOME/.jhalfs directory (or ./.jhalfs if $HOME is unset), runs menuconfig as needed, symlinks all the scripts included in the package to the .jhalfs dir and then invokes .jhalfs/jhalfs.sh run

To test this locally without building a package you can run pip install -e .

To run the unit tests, first pip install tox then run tox.

jhuntwork commented 5 years ago

@pierre-labastie I consider this a proposal - if you don't like this direction we can stick to just the shell scripts.

pierre-labastie commented 5 years ago

Well, I must first tell I've never programmed in Python... I can read it, but not produce code, although I'm confident I can learn within a reasonable amount of time (specially if I have to write some code myself).

So do not expect my review to be very accurate... I'll trust you for being able to produce "well written" Python. I have a practical questions, though (so you can see how dummy I am with Python...):

Do I really need to install to system? What I like with the present tool is the possibility to git clone the anywhere, type make (after setting the LFS partition of course), fill the form, and watch the run. IIUC, with the new system, I'll need to install the package (whether with setup.py or pip) after cloning the repo, then type "jhalfs", fill the form, and watch the run... Or can I just run jhalfs without installing? (I do not think so, since there is the dependency on kconfiglib). Also, with pip or setup.py, kconfiglib would be installed into the system python directory, or am I wrong here? If you could find a way such that everything remains in or below the cloned directory, I'd prefer that.But maybe I'm to short-minded.

jhuntwork commented 5 years ago

So, yes, in one sense, you do need to 'install' it. It's a feature that helps make sure any dependencies are also present and installed.

However, if you want to install it to a non-system directory you can do:

pip install virtualenv # If virtualenv isn't already on the system, sometimes it is
virtualenv ~/jhalfs-env
source ~/jhalfs-env/bin/activate

This creates a special local setup in ~/jhalfs-env that has a copy of the python binary, pip and a new, completely separate python library path. This means you can then use pip to install any tools you need to a non-system location.

And then if you want the latest released version from pypi.org (assuming we've published there):

pip install jhalfs

Or if you want the local git clone

pip install -e .

So, yeah, it's a slightly different way of working.

SheepReaper commented 5 years ago

The non-python/pip analogue would be creating a nuget/npm/psgallery published package that anyone can download from a terminal or as a dependency for something else and run as is.

On Thu, Apr 25, 2019, 5:36 PM Jeremy Huntwork notifications@github.com wrote:

So, yes, in one sense, you do need to 'install' it. It's a feature that helps make sure any dependencies are also present and installed.

However, if you want to install it to a non-system directory you can do:

pip install virtualenv # If virtualenv isn't already on the system, sometimes it is virtualenv ~/jhalfs-envsource ~/jhalfs-env/bin/activate

This creates a special local setup in ~/jhalfs-env that has a copy of the python binary, pip and a new, completely separate python library path. This means you can then use pip to install any tools you need to a non-system location.

And then if you want the latest released version from pypi.org (assuming we've published there):

pip install jhalfs

Or if you want the local git clone

pip install -e .

So, yeah, it's a slightly different way of working.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/JHALFS/pull/1#issuecomment-486846735, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLQ6JPF2IGJ6OWXK5B2LRTPSIP5JANCNFSM4HIQNIZQ .

pierre-labastie commented 5 years ago

Also, before leaving, one more thought: we have users who have been trained to use jhalfs in the same way for more than ten years. We should take care of not piss them off too much. If we tell: now, you need to install the thing, change to xxx dir, type "something new", and you need to reinstall each time there is something new on the github repo, I think they could get angry. OTOH, I understand the idea of hosting it on pipy.

I'm all for going to python, I'm glad I can learn it, but I'd like to have a way to have everything in the cloned dir. That does not prevent having setup.py to install it on the system (BTW, a true LFS'er should be reluctant using pip), but I think that, at least for a transient period, we should keep the ability to run without installing (that may amount to just reinstate the menu sub directory).

Sorry for being a little conservative, but it's not a new project, and we have users.

pierre-labastie commented 5 years ago

Yet another thought: if we go to python, I think we should have something like a "book object", with associated methods (parse, set parameters, possibly check). Those methods could be implemented differently depending on the type of the book (LFS, CLFS, BLFS, new layout..). But we would have a unique API to the main code.

Does it sound feasible?

SheepReaper commented 5 years ago

Why not go all the way? Host an api somewhere that lets you GET the code for X chapter formatted in whatever language you want.

Then jhalfs client code turns into: for package in packages do: get code from api; run code;

The api would be responsible for parsing the book. (Because the api hides the logic from the end user, you could consider this the language-neutral part, and have light-weight client code in whatever language you want)

It could parse on demand, but this would be slower than say, parsing the book only when the book updates and storing the output for faster delivery...

Seems more automate-ey...

On Fri, Apr 26, 2019, 3:58 AM Pierre Labastie notifications@github.com wrote:

Yet another thought: if we go to python, I think we should have something like a "book object", with associated methods (parse, set parameters, possibly check). Those methods could be implemented differently depending on the type of the book (LFS, CLFS, BLFS, new layout..). But we would have a unique API to the main code.

Does it sound feasible?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-486965033, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLQ6JLBUBHAL5ZDXSAJBXTPSKYYTANCNFSM4HIQNIZQ .

jhuntwork commented 5 years ago

@pierre-labastie - I understand your reservations about 'installing'. I think I might be able to adjust the Makefile so that the basic feel is the same for those that want to use it like that. I'll take a look at that.

As far as objects go, absolutely, I agree. In fact, you can possibly have a 'base' book object and then all the individual book objects inherit that.

@bryan5989 - Web APIs are great, and that's definitely an interesting idea... might have to put that on the back burner for the moment, but it is interesting. Providing a service would bring its own new set of challenges.

jhuntwork commented 5 years ago

OK, I updated the Makefile now too. Now the behavior (when using make) is similar to the previous versions. It does require pip3 being installed on the OS, though. Also it is still fully compatible with being packaged and hosted on pypi.org and installed to the system normally via pip, too.

pierre-labastie commented 5 years ago

I've been able to try this, at last... On a debian machine (jhalfs is supposed to be run on a host machine):

Note that, if you just clone the pythonize branch, there is no pkgmngt in the top directory, but this means the doc in the README.PACKAGE_MANAGEMENT has to be modified. More importantly, it means our users habits would have to be modified!

Now I see you have not moved the custom directory under jhalfs. Looks like an oversight, but actually, it may show us the way to go: keep sub-directories that the user may have to modify under the top level directory: those are optimize, pkgmngt, and custom. I guess this means changing the code in some of the functions that copy those directories (or parts of them) to $JHALFS_DIR, but it would prevent changing our users' habits.

Tell if you'd like me to do the necessary changes.

jhuntwork commented 5 years ago

All those suggestions sound good. Since you know what needs to be done, it's fine with me if you want to add a commit to this branch. It should become part of this pull request.

pierre-labastie commented 5 years ago

Now I see you have not moved the custom directory under jhalfs. Looks like an oversight, but actually, it may show us the way to go: keep sub-directories that the user may have to modify under the top level directory: those are optimize, pkgmngt, and custom. I guess this means changing the code in some of the functions that copy those directories (or parts of them) to $JHALFS_DIR, but it would prevent changing our users' habits.

Tell if you'd like me to do the necessary changes.

Well, because of the symlinks in .jhalfs, the change is not in the shell scripts, but in the init.py file! I've tried to do it, but it takes time. Python is not complicated, but there are sooooo many libraries to learn...

I'll commit what I have done (not a big change, but I am not sure it is good pPython coding), feel free to amend.

func_install_blfs is broken, though (it wants to move the menu directory so that it can be used by the blfs tools, but this directory does not exist anymore). Looks like the whole virtualenv build should be done again for blfs tools... Also init.py should be copied, etc. I fear to embark into some ad hoc changes, without taking the time to think of the big picture...

jhuntwork commented 5 years ago

OK, no problem. I'll review what you have and possibly add some tweaks. I won't be able to look at it for a few hours though.

SheepReaper commented 5 years ago

Feel free also to create personal branches for WIP. You can do squash commits to a feature branch (pythonize in this case) with an open pull request and they'll show up on the pull request.

This comes in handy when multiple peeps are working on the same feature. Also cuts down on unbuildable/unrunnable commits.

This will become more obvious with a CI/CD integration, you don't actually want to send *every commit to CI, especially WIP commits.

Sorry I'm not much help at the moment, been on the road mostly.

On Mon, Apr 29, 2019, 1:26 PM Jeremy Huntwork notifications@github.com wrote:

OK, no problem. I'll review what you have and possibly add some tweaks. I won't be able to look at it for a few hours though.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-487669754, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLQ6JI2K33FEKKAMHI4QO3PS4VVHANCNFSM4HIQNIZQ .

pierre-labastie commented 5 years ago

Thinking more about this, I have the impression we are going into a dead end: it's very good to go Python, but we should just try to do an independent python-jhalfs, rather than trying to be compatible with old scripts. Otherwise, we'll spend more time trying to be compatible than trying to do a good job with Python... The old scripts do more or less their job, and should not need to be modified for the sake of pythonizing. We should rather put our energy in designing a good python API, on one hand, and maybe in improving the old scripts capabilities (specially regarding BLFS) on the other hand.

So my proposition is to keep the pythonize branch, for implementing the python API, and not to merge it to master. When it is ready, just have the pythonize branch become master, and get rid of the old scripts... Actually, it could even be a different repo.

Keeping open for one day more, for more comments, and unless a good argument is found for merging the PR, I'll close it.

jhuntwork commented 5 years ago

OK, that sounds fine. That will allow the python work to flow a little more naturally. I was trying to keep it from being too dramatically different given how conservative the core audience tends to be, but I think tackling it as a new, different tool may make sense.

SheepReaper commented 5 years ago

Unrelated but related, I started assessing what a pick-your-language api server might look like. And struggling a little with the docbook format. Part of me says: it's just xml dummy, parse it. But I'm running into issues with the non-xml entities. I want to just import the .ent files as xml, but I'm probably going to have to write a parser/transformer. The xml library I'm using won't read files without the xml doctype/dtd.

I'm challenging myself to see if I can make this work without modifying the existing files.

Any suggestions?

C# .net std native xml libraries.

On Tue, Apr 30, 2019, 11:35 AM Jeremy Huntwork notifications@github.com wrote:

OK, that sounds fine. That will allow the python work to flow a little more naturally. I was trying to keep it from being too dramatically different given how conservative the core audience tends to be, but I think tackling it as a new, different tool may make sense.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-488001113, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLQ6JM5MMC6RCZJQABXJ3TPTBRLTANCNFSM4HIQNIZQ .

pierre-labastie commented 5 years ago

Unrelated but related, I started assessing what a pick-your-language api server might look like. And struggling a little with the docbook format. Part of me says: it's just xml dummy, parse it. But I'm running into issues with the non-xml entities. I want to just import the .ent files as xml, but I'm probably going to have to write a parser/transformer. The xml library I'm using won't read files without the xml doctype/dtd. I'm challenging myself to see if I can make this work without modifying the existing files. Any suggestions? C# .net std native xml libraries.

If you want to parse the .ent files as xml, you need to have them read as external entities from an xml file. The shortest such file is:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE bridgehead PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
   "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  <!ENTITY % general-entities SYSTEM "/path/to/general.ent">
  %general-entities;
]>
<bridgehead/>

But actually, you'll have not much more. What do you want to do exactly?

Sidenote: we'll need to discuss what we want to do; where is the best place?

ulfalizer commented 5 years ago

Kconfiglib guy here,

https://github.com/automate-lfs/jhalfs/commit/a645174fd43ba4eee84089965df85785878e7aa6 made me realize that it's a bad idea to special-case non-dotfiles when saving $KCONFIG_CONFIG.old, so I changed it: https://github.com/ulfalizer/Kconfiglib/commit/4fed39d9271ceb68be4157ab3f96a45b94f77dc0

Just as an FYI, in case you update Kconfiglib later. I bumped the major version as well.

SheepReaper commented 5 years ago

@jhuntwork Is is simpler to start python-jhalfs from scratch or modify the existing jhalfs?

I'm suggesting we fork the repo (new repo in the org) and move this branch there...

I can set up the same layout (projects And the other repo stuff)

On Tue, Apr 30, 2019, 4:45 PM Ulf Magnusson notifications@github.com wrote:

Kconfiglib guy here,

a645174 https://github.com/automate-lfs/jhalfs/commit/a645174fd43ba4eee84089965df85785878e7aa6 made me realize that it's a bad idea to special-case non-dotfiles when saving $KCONFIG_CONFIG.old, so I changed it: ulfalizer/Kconfiglib@4fed39d https://github.com/ulfalizer/Kconfiglib/commit/4fed39d9271ceb68be4157ab3f96a45b94f77dc0

Just as an FYI, in case you update Kconfiglib later. I bumped the major version as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-488108889, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLQ6JK7I4EZZMWIA4Q4XHTPTCVVZANCNFSM4HIQNIZQ .

jhuntwork commented 5 years ago

Maybe let’s start a new blank repo. I can commit a first version of the base code.

Opportunity here for name suggestions too.

On Tue, Apr 30, 2019 at 18:49 SheepReaper notifications@github.com wrote:

@jhuntwork Is is simpler to start python-jhalfs from scratch or modify the existing jhalfs?

I'm suggesting we fork the repo (new repo in the org) and move this branch there...

I can set up the same layout (projects And the other repo stuff)

On Tue, Apr 30, 2019, 4:45 PM Ulf Magnusson notifications@github.com wrote:

Kconfiglib guy here,

a645174 < https://github.com/automate-lfs/jhalfs/commit/a645174fd43ba4eee84089965df85785878e7aa6

made me realize that it's a bad idea to special-case non-dotfiles when saving $KCONFIG_CONFIG.old, so I changed it: ulfalizer/Kconfiglib@4fed39d < https://github.com/ulfalizer/Kconfiglib/commit/4fed39d9271ceb68be4157ab3f96a45b94f77dc0

Just as an FYI, in case you update Kconfiglib later. I bumped the major version as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-488108889, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABLQ6JK7I4EZZMWIA4Q4XHTPTCVVZANCNFSM4HIQNIZQ

.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/automate-lfs/jhalfs/pull/1#issuecomment-488143259, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB2QCXVYDPKUJDBFJNQSNLPTDEJHANCNFSM4HIQNIZQ .

pierre-labastie commented 5 years ago

Maybe let’s start a new blank repo. I can commit a first version of the base code.

Don't you want to discuss more about the code structure first? I'd like to avoid repeating what has been done for jhalfs: start small and add things, which then leads to big unmaintainable code...

I'd rather do something like:

  1. identify what we need: kind of books we want to treat, what we want to do with them, what we want to generate: only raw scripts or decorated scripts (for example with package management, with package download and unpacking instructions, with instructions for becoming the root user,etc)
  2. Once we know what to do, think of a data structure for the books, which allows accomplishing the tasks above
  3. Code back ends to transform our data structure into what we want.
  4. Code front ends to transform a given book file into our internal data structure

Note that 1 and 2 may overlap in time, as well as 2 and 3, we can even go back and forth, if something is too difficult to implement. Once we are confident with 2, we can also begin 4 for sample books (maybe small books for tests, or existing books)

Opportunity here for name suggestions too.

plalfs (for python language automated linux from scratch, of course :wink: ). More seriously, why not python-alfs?

jhuntwork commented 5 years ago

OK. I think starting small and adding things over time can be fine, if you maintain a strong sense of goal and purpose and you can make sure that contributions closely adhere to that. I kind of failed to do that last time with jhalfs. Inexperience and other life factors led to me conceding control of the project early on. Even though there were certainly many useful and nice contributions, I think that kind of escalated into something difficult to maintain.

So, yes, if you want to have a larger discussion around scope and design that's fine. It might be good to take that discussion back to the alfs-discuss list so those lurking there can contribute if they want. Or maybe an issue here would be fine too. Probably best to just close this for now though.