How feasible is to create an alternate pet archive format and make it the default one?

wdlkmpx commented 5 years ago

Basically a regular tarball or maybe a sfs, with these possible extensions:

.pet.tar.gz .pet.tar.xz

*.psfs

With optional sha256 sum

Following arch linux's *.pkg.tar.xz example.

Without md5 and archive name inside the pkg, with the usual stuff. pet.spec pinstall.sh, etc.. a package that it's easy to manipulate, extract and create in any distro, but with a special meaning (MIME pkg) for woofce.

This involves making a few changes to woofce scripts, i actually have dir2pet already "converted", with a few lines.

The scripts that install stuff, unpack stuff, should just recognize a new .pet format which requires less hacks

Not alternate, the new default pet archive format. It's just the .pet files disturb me i can no longer hide it..

s243a commented 5 years ago

As long as there is a conversion tool that on can use to convert to the old pet format than it whouldn't be an issue. If puppy is to go with a new package format, may I suggest putting all the files to be installed in a separate directory than any installation scripts or meta data.

wdlkmpx commented 5 years ago

I mean an archive format with the same stuff and specifications, but without

embedded md5sum
package name as a directory.

With some minor changes i guess it can be converted to a .pet with tgz2pet (supports .xz .gz)

That's why this requires very small changes to the relevant scripts, specifications do not change, only the archive "format", the resulting pkg, can be extracted, edited, repacked using any distro without the need for special tools

wdlkmpx commented 5 years ago

I guess it's not really feasible, this change, along a ton of other changes should have happened in 2013

01micko commented 5 years ago

I think it should be done. There is really no specification for the PET package format. It doesn't make sense.

All other formats simply expand to '/' and extra processing done with postinst, doinst or whatever. Sure, people will moan but they will get over it. Changes like this would have to be 'grandfathered' over a period.

Also, petspec could do with more fields. here's Barry's spec:

pkgname|nameonly|version|pkgrelease|category[;subcategory]|sieize|path|fullfilename|dependencies|description|compileddistro|compiledrelease|repo|

A few of these are actually unused in a pet. They may be used when flattening a distro's Packages* file.

So, who is up for the challenge?

wdlkmpx commented 5 years ago

I'm not good with terms, when i say specification, i mean the pet.specs, pinstall.sh, puninstall.sh... the usual files that are added to a pet pkg.

I just want to change the 'archive format' from a corrupted tar file to a normal tar file and a normal package for installation with .pet.tar.xz/.pet.tar.gz extension.

It really takes a few minutes to make all the required changes, but it requires a new build for everyone to produce the same packages.

Of course .pet will still be supported, it's only that the processing of the new tarball will skip some really awful hacks, i guess some simple stuff can be implemented in zwn to deal with this, some shadow packages that are not recorded as installed.

That way unification is less dramatic, stuff is compiled for the .pet pkg dbs, and i certainly won't a create .deb packages or something.

wdlkmpx commented 5 years ago

At this point in time, i'm really used to no use a package manager. I actually used the ppm or any other package manager very few times. And most of them just to test code changes.

I assemble my own sfs files, i compile everything else i need. I use wine. Due to the way puppy works i was forced to be this way.

I basically create my own world.

But i think i can provide some updated packages to the repos, i don't focus in just a single compat distro version, i see the big picture.

But it should happen with *.pet.tar.gz/xz files.

wdlkmpx commented 5 years ago

The 13th field: checksum... but only in the package databases, to be verified by the build scripts and the ppm. And probably other fields.

I remember my brother using arch linux, he could open the packages with file-roller, easy to explore the contents. In fact when he clicked on the package file-roller opened it

It was an option from the drop down menu which installed the package.. to me that seems just right.

To this day, arch and slackware use scripts to produce their packages.. i think. But i still don't know what they do with the checksum stuff. I've no idea. Their packages are standard tarballs, i can open them with engrampa and explore the contents.. and extract the files i want, or delete/add files without manually repacking.. i can't do that with a corrupted tarball.. the archive managers refuse to open it.

This change can be considered philosophically correct, but i kind of feel ashamed to implement it hahaha.

You may wonder why does a dog bark.. yes, it's to warn you, he's telling you he'll shred you to pieces so you must go away from his property.. yes, that's what greet you on the first boot.

If you see it with other eyes, the dog is probably in distress and its barking means you should call PETA to denounce abuse in the puppy distro.

I did change that distressing sound, but it was restored.

rizalmart commented 5 years ago

I have a suggestion just pack it as a tarball without package name as directory and md5 checksum (forget about the md5 thing) just like arch at slackware package. Also retain pet.specs to retain backward compatibility. In terms of file extension arch linux style (*.pet.tar.xz) is the best because it was easily accepted by earlier ppm. The compression setting for the alternate pet package was always set at best compression for smaller filesize

Here is the proposed new pet archive content for example: foo.pet.tar.xz

/etc /usr pet.specs pinstall.sh puninstall.sh

You will see here that pet package specification remain intact but the folder package name was removed and md5 thing was dropped

wdlkmpx commented 5 years ago

I reject that suggestion..

rizalmart commented 5 years ago

What is the reason for rejection? Is there something wrong?

wdlkmpx commented 5 years ago

The previous comment was sent from email, and i see the message has changed.

That suggestion is what i propose, you're repeating what i say, perhaps thinking it's a new idea.

rizalmart commented 5 years ago

I got your point now.

wdlkmpx commented 5 years ago

The compression defaults to xz, but can also be gz.

That sounds fair, what doesn't sound fair is to delete basically all of the user configurations at shutdown.

dir2pet supports switches, i have tested it with this proposed stuff.

ghost commented 5 years ago

I can open a .pet file if I rename it to .tar then click on it (opens with xarchive) then I can extract it. I am not trying to detract from this new format though, dir2pet does seem to take too many steps (but I think there was a parameter to skip some steps, -s? might have to look at that again). :smile_cat:

wdlkmpx commented 5 years ago

I use engrampa, i only use xarchive when engrampa is not installed

I cannot open it with engrampa:

Xdialog_DOC-2015.08-i686_stretch.tar.xz

xz: (stdin): Compressed data is corrupt tar: Child returned status 1 tar: Error is not recoverable: exiting now

ncurses-5.9-i486-4.txz

OK!

lxtask-0.1.9-1-x86_64.pkg.tar.xz

OK!

I don't know if any checksum stuff is added to those packages, but it certainly doesn't corrupt them..

wdlkmpx commented 5 years ago

I must say that xarchive is certainly unique.. performs operations even when the tools it calls fail, all the other archive managers check the exit code

from wrapper_cmd: /usr/lib/xarchive/wrappers/tar-wrap.sh -o /mnt/sda3/Xdialog_DOC-2015.08-i686_stretch.tar.xz 
xz: (stdin): Compressed data is corrupt
tar: Child returned status 1
tar: Error is not recoverable: exiting now
wrapper exited with: 0

With xarchiver, another primitive gtk2 app, i see this:

An error occurred!

Please check the 'Store archiver output' option to see it.

It's worth mentioning that when one of the advanced archive managers is installed, pupzip (the defaultarchivemanager unless you specify another) will use the advanced one:

https://github.com/puppylinux-woof-CE/woof-CE/blob/testing/woof-code/rootfs-skeleton/usr/local/bin/pupzip#L53

peabee commented 5 years ago

I can open a .pet file if I rename it to .tar then click on it (opens with xarchive) then I can extract it.

I do the same - just change .pet to .tar.gz (or xz) and then click to open..... very easy.....

wdlkmpx commented 5 years ago

I think peebee's lxpup only includes xarchive, i was shocked and considerably angry when i found that, it doesn't make sense, well it's peebee's creation

We all use what we think it's best, and craft the distro we want.

So, this implementation may indeed happen, but it will probably not be the default package format created by dir2pet, but it will be used by me.

And no one can force to create packages with a format i hate, they will either accept my *.pet.tar.xz packages or.. not.

radky2 commented 5 years ago

After a slight modification of tar-wrap.sh, xarchive will support loading/viewing/extracting gz and xz compressed pets.

Go to /usr/lib/xarchive/wrappers/tar-wrap.sh and add the pet extention to GZIP_EXTS and XZ_EXTS:

Supported file extentions for tar TAR_EXTS="tar" GZIP_EXTS="tar.gz tgz pet" BZIP2_EXTS="tar.bz tbz tar.bz2 tbz2" COMPRESS_EXTS="tar.z" XZ_EXTS="tar.xz txz pet" LZMA_EXTS="tar.lzma tar.lz tlz"

rizalmart commented 5 years ago

I wonder is there any way to remove the md5 checksum string append at the end of tarball binary data using bash?

wdlkmpx commented 5 years ago

Back in 2014/2015/2016? i used to use xarchive, and added code / fixed the wrappers as i updated the the relevant tools.

But.. i ended up deleting all that code when i switched to engrampa, i could have contributed that code to the petbuilds or something, but at that time i wasn't quite aware of the petbuilds, i guess anybody can attempt to implement stuff and provide the updates.. i guess the latest xarchive/p7zip should be added to the old woofce distros. I don't remember what i did, or if i did contribute something.

I'm updating my gtk2 engrampa fork to 1.23.0w, need to push the updates and create a release tarball.

I wrote many scripts and also deleted many of them, and i guess i'll delete some more, when i find something better that is still small or fixable i tend to delete stuff.

That's why people should not fear if i send their code to the shadow realm

peabee commented 5 years ago

After a slight modification of tar-wrap.sh, xarchive will support loading/viewing/extracting gz and xz compressed pets.

The xarchive petbuild could be updated to include this simple change......

peabee commented 5 years ago

Ohhhhhhhhh - the xarchive petbuild seems to have been deleted?????????

mavrothal commented 5 years ago

I mean an archive format with the same stuff and specifications, but without embedded md5sum package name as a directory.

Late to the party 👅 ... These 2 were dictated historically by the loose character of puppy. Pets are distributed through many avenues so you need something to validate the download. Formal distros have a list with the hashes of the packages and the package manager does the validation against that. This could also be done with the "*-official" package lists but In puppyworld you embed the hash so you can validate the download regardless of its origin. Without it you have find another way for validation (unless you want ONLY official packages used 😮 ). The directory thing is a convenience/workflow relic. make > new2dir > dir2pet. Also makes it easy for hand-crafted packages by the "noob next door". It is not the standard but I fail to see the harm. Actually makes it very easy to open and extract the contents of a package ( I made an "app" in the Mac that does that, so I'm surprise that you have problems with that). Of course you can do whatever you want but would think that use-case rather than personal bias should be the driving force behind changes (I have a cat but I do not mind barks at all 😛 )

s243a commented 5 years ago

Regarding the md5 hash, if we wanted a real strong authentication then the hashes should be signed with pgp signatures, and we should allow more than md5 hashes. I recommend sha256 as the standard given that if you have enough computing power you can fake the md5 hash.

Regarding the nested directory structure created by dir2pet, where the package name is a directory inside the archive file; I believe that this wasn't part of the original design of the ".pet" format but rather a bug that was introduced. I'm pretty sure in the original pet format that this nested directory structure didn't exist because pet packages that have this structure don't work on older version of puppy and some package conversion tools (e.g. tazpkg) don't work when a pet has this nested directory structure.

wdlkmpx commented 5 years ago

Looking for fresh .efi files, i found .rpm files that busybox rpm (and the old exploderpm script) cannot recognize.. compressed with Zstd.

I guess only the full rpm tool can handle that, and that app including deps is... big.

pupzip does something really crazy, extracts the rpm file and creates a tar file. So that it can be opened with $XARCHIVE. That's completely and utterly wrong. Stone cold crazy you know.

A few weeks ago i downloaded a password protected zip file, apparently it's a special zip file, i could not open it with any linux tool, running 7-zip 19.00 with WINE i was able to view and extract the contents

There's plenty of work to do.

01micko commented 5 years ago

Slackware -current ships with zstd. Squashfs also supports it - https://facebook.github.io/zstd/

I suppose it is becoming standard for better or worse.

Some of the puppy 7zip progs are quite old. Perhaps it should be updated and put in the common repo?

Anyway, back to the main topic. I'm all for it but with rules, strict rules that the packaging tool (whatever that may be) enforces so that we have consistency.

dir2pet allows pet creation with out a version number! That is bat excreta crazy!

Some pets are like this - mygame_0.1_x86.pet
Others like this -mygame-0.1-x86

Proposal

package names can only have lower alpha/numeric characters with perhaps underscore and/or hypen (many source packages have an underscore or hyphen in the name)
package versions must lead with a digit, can include a period, underscore, alpha but no-hyphen !
all packages must include an arch field at the end. eg: x86_64, i686, armel, armhf, armv6, armv7, noarch, all, any etc.
post install, post remove and meta data goes to a separate directory, similar to slackware's install/doinst, let's say pinstall/pinstall.sh. Irrelevant if it's executable as it should be called with sh pinstall/pinstall.sh at / anyway. All paths in an install script must be relative and nothing should install to root,run,var/run ortmp`. All code should be POSIX and try to use pure shell as much as is practical.
summing that up, the package should have 3 fields à la debian. $PACKAGE-${VERSION}${EXTRA_CRUD}-{$ARCH}.${extension}, however unlike debian we use hyphen "-" instead of underscore "_". Slackware uses 4 fields with hyphen, $PACKAGE-$VERSION$-$ARCH-${TAG}_${EXTRA_CRUD}.t[lbgz]z. So quite easily with rev and read we can sort out the package fields. Also, with this format we can quite easily convert a puppy package to a debian or slackware package that will play nice with their respective package management. More on that down the page.
sha256 must be appended to the database specs
the packaging tool should check all of this and generate a sane database entry once the package is compressed to final form.

The $extension is certainly up for debate but I'm leaning towards KISS. Any or all of these points are certainly open for discussion but I think the ability to easily convert the packages to another format is appealing.

For example, a debian package, for those who don't know, is actually an ar archive. When you run ar x some-package_0.0.1_all.deb you are left with 2 files .. control.tar.gz which contains any meta data and post installation/removal files and the debian package version, debian-binary, currently at 2.0. The other file data.tar.[gx]z contains the program which can be extracted at / if you want to install it quick and dirty. So to convert the proposed puppy package to a deb just extract the puppy package into a directory, pull out the meta and pinstall.sh and convert the meta to a debian style control file, and pinstall to a debian postinst adding the debian-binary file containing 2.0 and tar those up as control.tar.gz. Then tar the rest up as data.tar.gz (or xz) the archive those with ar adding the deb extension replacing the hyphens with underscores in the package name. No acrobatics needed.

Slackware is even simpler. Simply extract the puppy package, rename pinstall/pinstall.sh to install/doinst, optionally convert the meta to slackdesc or delete it, then tar it back up with a -$TAG (-PET maybe?) description so it has 4 fields and a .t[gx]z extension. Done. Less acrobatics.

Let's get to it shall we?

wdlkmpx commented 5 years ago

I can update the (official) older compat distro pups, so that everything is in sync.. that's not a problem, the problem is uploading stuff to ibiblio. Perhaps it should be discussed privately.

Days ago i implemented what i proposed, it works in petget and misc utilities, the build system. But i didn't test with the ppm. It's just an archive format change, the "specs", aka pet.specs, pinstall, puninstall remain the same.. the results are identical when processed by the woofce scripts.

What you propose sounds right, and it's a more drastic change, regarding the extension if we're going to keep it simple, then it can't be .pet, perhaps pxz/pgz, because it's another format.

I think debiandog has a tool to convert .pet to .deb.

To be continued

01micko commented 5 years ago

About extension.

Originally I liked the .pet.tar.[gx]z idea for simplicity (like arch) , or even .pet.t[gx]z (like slackware) but maybe keeping .pet isn't a bad idea, like .gz.pet or .xz.pet or even create a whole new thing like gpet, xpet or even zpet (if we adopt zstd :stuck_out_tongue: )

There are a lot more than @wdlkmpx and myself subscribed to this thread. @peabee @s243a @ninaholic @mavrothal @radky2 @rizalmart speak now or forever hold your peace!

I'd like to hear the opines of @mrfricks @zigbert @technosaurus @dimkr and anyone else interested.

peabee commented 5 years ago

I understand that the .pet format may not be 100% perfect..... but it seems to have "worked" "fairly well" for many years...... Any change should be subject to wider debate and comment on the forum by "users" as well as "devs"..... But a clear description is needed as to what exactly is being proposed, what changes would arise as seen by "users" and what impact the changes would have on the 1000's of .pets in existence (not just on ibiblio)....... So as long as the proposal is for a new format to run alongside .pets but not to replace .pets then I could support that, but if the proposal is that new puppy builds would only support the new format and would consign all existing pets to history and the trash can then I wouldn't be able to support that.

01micko commented 5 years ago

I understand that the .pet format may not be 100% perfect.....

Nothing is perfect

Any change should be subject to wider debate and comment on the forum by "users" as well as "devs".....

Sure, why not? Anyone can start a discussion wherever they like.

But a clear description is needed as to what exactly is being proposed, what changes would arise as seen by "users" and what impact the changes would have on the 1000's of .pets in existence (not just on ibiblio).......

https://github.com/puppylinux-woof-CE/woof-CE/issues/1494#issuecomment-515717101

Not clear enough? It may not quite be 100% complete but it's not far off. I intend to write up something like an RFC for the spec. That should be distributed in all future puppy distros and a page dedicated to it on puppylinux.com and the wikka . I'll also help with tools to package and install if needed.

Of course, you being a "dev" know that change takes time and it can be either a short time, that is kill all old pets and only accept new, or a long time, as in "grand fathering" the change until everyone is used to it. Of course we lean to the latter!

So as long as the proposal is for a new format to run alongside .pets but not to replace .pets then I could support that,

It has to replace the current format eventually. That is the point. The current .PET format is awful. Barry may have had his reasons but IDK what they were. It is slow and cumbersome. Unpacking a package at /root and moving it to install it? Not much thought went into that when the proposed new format can simply unpack at /.

but if the proposal is that new puppy builds would only support the new format and would consign all existing pets to history and the trash can then I wouldn't be able to support that.

Well, not going to happen. However, I feel that if a new format is taken up then the old format (for packaging) should be deprecated immediately. No deprecated package should be uploaded to the repos after the cut off point. However, old format pets should still be able to be installed, albeit with an appropriate warning.

We need to grow. Puppy has been stagnant for too long. I know I have been quiet and haven't helped much but my reasons are personal. This may be the kick up the bum that puppy needs!

01micko commented 5 years ago

@rizalmart

I wonder is there any way to remove the md5 checksum string append at the end of tarball binary data using bash?

Not a pure bash or shell solution but sed, awk or dd are all capable and probably some others.

@ninaholic

(but I think there was a parameter to skip some steps, -s? might have to look at that again). :smile_cat:

# dir2pet -h

Usage: dir2pet [option]... [<name-of-dir>]

Create a compressed file containing the content of a directory

Options:
  -h, --help               show this help
  -C, --change-compression-default Change the default compression
                   (gz or xz, currently gz)

--------------------------------------------------------------------

Basic:
  <name-of-dir> the directory, not path, to be packaged.
        THIS IS THE PACKAGE.
        (Omit if specified with -p option)

  EXAMPLE:
    # dir2pet my_fun_game-0.1-i486

--------------------------------------------------------------------

Advanced:
  -s           skip all questions - for package build scripts
  -p=<name-of-dir> directory to be packaged. THIS IS THE PACKAGE.

The following options are only useful with the "-s" option.
For best results, use as many as possible:

Compression: (use only one, default if none chosen is \"$comp\")
  -x    use xz (high) compression
  -g    use gz (low) compression

Non-compulsory:
  -k      keep pet.specs / do not overwrite it
  -n                    no distro information
  -i="<repo> [version]" override default distro info
                        -i="noarch"
                        -i="debian stretch"
                        -i=""
  -w="<description>" quoted description of the application
  -d=<+deps>         comma-delimited dependencies, with prepended
             "+" sign
  -c=<Category>      category of application, for package manager -
             allowed categories:
            BuildingBlock (default)
            Desktop
            System
            Setup
            Utility
            Filesystem
            Graphic
            Document
            Business
            Personal
            Network
            Internet
            Multimedia
            Fun

  EXAMPLE
    # dir2pet -x -s -w="fun game" -d=+gtk+,+ffmpeg,+cairo -p=my_fun_game-0.1-i486

peabee commented 5 years ago

1494 (comment)

Not clear enough?

Not really - that's a spec of a new format....what is needed is something much more philosophical as to why we need a new format, what aspects of pets its trying to improve, what the impact on a user will be, what the impact on a dev will be, and what will happen to the myriad pets in existence....etc.

peabee commented 5 years ago

oh - and why (like Fatdog has done) we can't simply "adopt" an existing packaging format....

wdlkmpx commented 5 years ago

When i changed the barks to a more pleasant sound, it was reverted.

Some puppy users freak out when they see partitions being listed in the gtk open/save dialogs. And many more strange and abnormal behaviors and thinking

I guess i can focus on updating and synchronizing the official pups for my benefit and somebody else's benefit.

s243a commented 5 years ago

@rizalmart

I wonder is there any way to remove the md5 checksum string append at the end of tarball binary data using bash?

Not a pure bash or shell solution but sed, awk or dd are all capable and probably some others.

@ninaholic In part I wonder if this is necessary since: "Both tar and cpio have a single purpose: concatenate many separate files to a single stream. They don't compress data. (These days tar is more popular due to its relative simplicity – it can take input files as arguments instead of having to be coupled with find as cpio has.)" https://superuser.com/questions/343915/what-is-the-difference-between-tar-vs-cpio-archive-file-formats

I suppose if one is extracting directly to "/" then such file integrity checks are helpful, but not not 100% secure since:

md5 hashes can be faked (sha256 is better)
the hashes aren't signeed by a gpg key.

If one first extracts the archive (instead to a temporary folder) then they can simply sort the files and pipe them to the hash algorithm. Also the package archive could contain meta-data where one or more people sign the authenticity of one or more verification hashes.

Perhaps the default settings should be such that one will only extract to "/" if one has external meta-data to verify the package and otherwise one will look for the meta-data in the meta-data folder within the package archive, prior to either extracting to root or copying the files to root. These security settings should probably be Overridable but some might consider that a secruity risk since then overriding these security settings will become an attack vector.

Not that while extracting to "/" is faster it can destroy symlinks. Therefore perhaps extracting to "/" shouldn't be the default behavior but might be an optional behavior if one wants faster installation.

P.S. ALso note that: "Both tar and cpio have a single purpose: concatenate many separate files to a single stream. They don't compress data. (These days tar is more popular due to its relative simplicity – it can take input files as arguments instead of having to be coupled with find as cpio has.)" https://superuser.com/questions/343915/what-is-the-difference-between-tar-vs-cpio-archive-file-formats

which might factor into peoples opinion on whether or not appending the hash to the end of an archive is a good idea.

As a final note it would be ideal if adding additional verification signatures to the hashes didn't change the hashes. This would make it easier for multiple people to sign the content within a package archive. The only way this will be possible though is if the files are first extracted to a temporary folder before being extracted to "/" or alternatively there could be a file format called a verified package which is an archive consisting of the verificaiton signatures as well as the package. One would first extract the verified archive to get the signatures and hashes and then if everything was okay, extract the package from within the verified archive to "/".

s243a commented 5 years ago

post install, post remove and meta data goes to a separate directory, similar to slackware's install/doinst, let's say pinstall/pinstall.sh. Irrelevant if it's executable as it should be called with sh pinstall/pinstall.sh at / anyway. All paths in an install script must be relative and nothing should install to root,run,var/run ortmp`. All code should be POSIX and try to use pure shell as much as is practical.

I think that it is okay to supply an alternative folder to extract to within the package provided there is a list within the package of the final file names. If one intends to extract to "/" than said file list isn't required. The reason to do this would be to avoid overwriting files (e.g. overwriting puppies ps script, with ps-FULL).

I know though that if people use the wrong extraction folder (e.g. /tmp ) this has caused problems.

Regarding using pure POSIX, I think that this should be a recommendation rather than a requirement. People should be allowed to modify the hashbang of the post install scripts. Perhaps this recommendation should be heavily considered when adding a package to the official repo.

My reasoning here is that I find bash a lot more convenient than sh because of the shorthand for process substitution (e.g. <(...) and >(...)) as well as possibly better array handling. If I need process substitution and use ash instead than do I have to use trap statements to close file descriptors? I suppose I could use temporary files instead but I find this messy.

Process substitution is very helpful when one want to access global variables within loops and doesn't want to create unnecessary temporary files.

gyrog commented 5 years ago

If it's going to be done, make it worthwile. i.e. not only get rid of the checksum and the internal package name, but also separate the meta files into a defined sub-directory, e.g. /PUPPY The extension can be anything other than '.pet' Note: tar can now extract without supplying the compression, so could try simply '.pet.tar' or '.ptar' i.e. leave all compression issues to tar.
To get rid of the checksum in a script, see /usr/bin/pet2tgz. head -c -32 /path/to/top-stuff-0.1.pet > /path/to/top-stuff-0.1.tar.gz or head -c -32 /path/to/top-stuff-0.1.pet > /path/to/top-stuff-0.1.tar.xz
How: We survived the old sfs files to sfs4 conversion, we can do this. a. Provide utilities to convert all currently supported package formats to this new format. b. Make new Puppies support only the new format, with instructions on how to convert old formats, or internally convert any old formats. c. For existing Puppies, add support for new format as an extra format, but leave support for old formats as is.
Extracting a tar directly to / could create problems, unless you can be absouletly sure that there are no symbolic links to directories in the file system. If your source contains a diretory that corresponds to a symbolic link in the local file system, you have just stuffed up your local filesystem. I'm currently testing a hacked PPM that respects any symbolic links to directories in the local file system, looking good so far. I expect to release it in the forum in the next couple of days.

gyrog commented 5 years ago

checksum removal command could be: head -c -32 /path/to/top-stuff-0.1.pet > /path/to/top-stuff-0.1.tar

mavrothal commented 5 years ago

late in the party once more, but I’m not really using puppy much these days so take the following having that in mind.

I think there is a forefront and a background issue here. The forefront is package specs and package management. The background is puppylinux future priorities and direction.

Lately we see increased conformity with mainstream linuxes, a focus on ease of development and occasionally, “because is better” changes. The JWM/ROX, trim to the max, root-only, Gtk2+, shell/gtkdialog, hack-as-you-go, layered OS, that is easy to use focusing on the linux newby holding-hands-in-every-step, is giving way to a highly configurable (ie needs knowledgable users going through lists of options), multi-user, multi window managers, increasingly gnome compatible, minimally if at all trimmed, increasingly well defined OS. I do not think that either one is necessarily better for the user but it should be clear where it is going and why. Kind of a SWOT analysis.

Puppy distinguished itself offering simpler, often unorthodox, solutions to various problems, that worked OOTB with minimal hardware requirements. Usability and use case was at the core of its philosophy. How would puppy repeat that in a conforming era? Which use-cases and/or develop-cases need to be addressed? (packaging for example is mainly a develop-case). Where is the priority (if prioritisation is needed)? Still interested in old hardware? CD boot/save (as size increases this becomes painful)? 64bit only? Still focussing on retirees? (User average age is 60+ according to an older forum poll). Are fed up XP/Vista users still a target (they do not exist anymore)? Should it be able to run say, an Apache server with all the goodies? Is it going to be updatable through packages (ie through the savefile/folder)? Should… ?(put your option here). You get the idea.

Higher conformity actually means that will be directly comparable with the big dogs and probably lose the remaining learn-to-hack puppy wildcats (that actually many current “conformist” stared as). So how would it be different than the live CDs of the main distros? Just because it can frugal install? What would the puppylinux niche will be in the Linux world of the 20s? It appears that there are now several persons that have some idea of puppy’s future direction. As long as they agree on some common goals and directions puppy could become a whole new animal. Either way, with explicit goals and directions things should be fine. In contrast, changes with implicit goals or even not thought out at all, have the danger of developing a Frankenstein that will eventually disintegrate.

In conclusion I think that Micko’s proposal as well as the changes pushing towards conformity are fine, but more important for the puppy developer than the puppy user. We have seen a flourish of puppies and puppy-likes lately but not necessarily puppy(like) users. Maybe this should be a concern or maybe just follow the “if you build it they will come” approach.
I believe that is time for Puppy to (re)define its goals and see how this will be best materialised, or at least start pushing commits with a use-case explanation 👅

01micko commented 5 years ago

@gyrog

a. Provide utilities to convert all currently supported package formats to this new format. b. Make new Puppies support only the new format, with instructions on how to convert old formats, or internally convert any old formats. c. For existing Puppies, add support for new format as an extra format, but leave support for old formats as is.

yes these are all valid ideas and similar to my thinking.

Extracting a tar directly to / could create problems, unless you can beabsouletly that there are no symbolic links to directories in the file system.

Yes. Any packaging tool/converter should pick up symlinks to dirs and reject.

@mavrothal

Old hardware is something that is fading. Many people find cheap tablets and phones are the way to go. Until recently my main dev box was a circa 2007 core2duo that shipped with vista! Slacko64 and even slackware proper ran perfectly fine on that. I think any distro would struggle on an old pIII or an old athlon-xp these days. Both of mine died an age ago. the cost of a new PSU outweighs the cost of a cheap dell or HP core 2 in decent condition. Yeah I can hear the sage's out there screaming "landfill!" but in all practicality these old boxes chew a lot of power with inefficient and old design. Should their use be encouraged?

I'd rather see (and am trying to do) leaps forward in puppy development in efficient arm designs such as rpi, odroid, cubie etc, older ones included. These are cheap and can run on an off the shelf battery for days or of a phone charger. Power consumption is very low.

So perhaps Puppy's goals should be redesigned slightly, which certainly doesn't stop anyone still supporting the old clunkers.

A redesign in package and management certainly wont stop that and as far as end user is concerned.. will they even notice? It's true, it mainly concerns developers.

wdlkmpx commented 5 years ago

Well, i guess, i guess, i guess... (reinserting disc)

After commit https://github.com/puppylinux-woof-CE/woof-CE/commit/9724cf500c202f6222202b335f997d1aba15ee75 zwn can download and "install" pet packages to the chroot dir

Here is an example https://github.com/puppylinux-woof-CE/woof-CE/commit/734ce98a40949e7296d42b1bd0ca845af30f1a37

And i guess i'll just abandon the idea, there's many things to do

01micko commented 5 years ago

And i guess i'll just abandon the idea, there's many things to do

Just do it without making it the default

Give everyone time to get used to it and iron out the bugs.

01micko commented 4 years ago

Here is an idea.

A pet package is basically a tar archive with the md5sum appended. This makes the tar archive corrupt.

How about a spin on the deb format?

create an ar archive with the same structure of a pet
get the sha256sum of this archive and print to a temp 'check' file
tar it all up (ar archive and the check file) with any tar compatible compressor (gz, bz2, xz, zstd - whatever) and name it somepackage-0.1-noarch-1.pat (or something, so long as it is consistent) and call it the PAT Puppy package archive (Puppy Archive Tarball). We pat our puppies, no? :dog2:

Of course the package manager must be adjusted to understand this and all puppies henceforth must include ar.

Advantages

more secure
more compatible with standard tools

Disadvantages

not backward compatible
must adjust package managers
slower packaging

jamesbond3142 commented 4 years ago

I didn't see this thread when it was originally posted, but now that you re-opened it, here is my 2 cents.

In FD, we moved away from PET because of the following reasons (very similar to the points mentioned above):

the embedding of md5sum make the tarball looks "corrupt" to tar, so it cannot be processed properly without "special tool" (basically cut out the md5sum before feeding it to tar)
the inclusion of the "package name" as the "directory" is unnecessary and complicates matters - what if the package is renamed? What does it package manager has to do if the directory-name inside the tarball doesn't match the tarball package name?
the petspecs is not suitable for longer package description names since everything is embedded in one long line.

My conclusion was: a) md5sum could be stored and checked in an external file, so the package becomes a normal tarball b) there is no need for "package directory", everything is based on "/" so that the package can be installed just by doing "tar -xf package.tar -C /". Package creation is just "tar -cf". c) use another scheme metadata that is a bit more flexible

And then I found out that Slackware package had exactly all of the above! (plus a nice and long-time tested package management tools too). I didn't think we needed to re-invent the wheel, so we adopted it instead of writing our own.

To mitigate compatibility problems, I wrote a "pet2txz" script plus ROX right-click action. We still have that today even though we left the PET format over 6 years ago.

Of course, it was not easy. In addition to the fact that people had hoarded PET packages, and teh fact that if we broke away from PET it also meant breaking away from a bajillion of existing Puppy packages, back in FD600 I spent a lot of effort to re-write the PPM from scratch, using gtk-server, splitting it cleanly into "CLI tools" and "GUI tools" (GUI tools does the operation by calling the CLI tools - so operations between GUI/CLI tools is always consistent) - with features that compete with gslapt. It worked very nicely, and it was backward compatible with PET format. So I became quite attached to it, and it was rather hard to let go. But eventually I forced myself.

I'm not advocating that puppy adopt Slackware package format. This is just some sharing from one who had done the switch before. And switching is not the end of the story either.

After we migrated, we realised that as flexible as the Slackware format was, there was something that it didn't support: it didn't support uninstall script (puninstall.sh). This was okay for a while, but some packages were really intrusive pinstall.sh and needed an equivalent puninstall.sh or the installation would left dangling files; so I eventually "expanded" the Slackware format with ability to run uninstall script (in a backward compatible way, if the same package is run by a genuine Slackware installation it would still be installed/uninstall properly - except that the uninstall script won't be executed). This is the price we pay for using someone else's package format. But overall, I think it works quite well for us.

Puppy's situation may be different. Unlike FD, Puppy can be built from many different sources. Whatever the package format chosen, these sources must eventually be "transformed" into Puppy's native format (even if it is only done temporarily and on-the-fly so that it can be installed and its files can be tracked). With an own package format you can create a format that is superset of the existing features; so converting to it is easy. If you choose to use an existing format (e.g. Slackware), it may not have the features of other, more advanced formats (e.g. deb, which supports package pinning, pre-/post- install and pre-/post- removal scripts, etc) which means that these will have to be "simulated" or "ignored.

Re: your proposed PAT format, the downside is mainly because: a) the fact that the checksum file is inside the tarball b) the need for double compression

(a) means that to check the checksum you need to extract files from the package. For a 100K package this is not a problem, for a 50MB package (firefox) this will take quite sometime. I propose the use of external checksum file (IIRC both slackware and debian use this strategy)

(b) means that to get a file listing you need to doubly-decompress the package - means more temporary space is needed during installation. I propose that you have a single compression, with files installed in "appropriate" "unused" location (e.g /install in Slackware format).

Of course, debian uses double-decompression (ar for the exterior format, and tarballs for data/control files), but then debian never set sight on running on less endowed systems. That being said, it's system is good enough to apparently run on a system as small as a Raspi, so probably my assessment above doesn't really hold enough water :rofl: .

gyrog commented 4 years ago

FYI I did produce utilities to support an alternate to ".pet" package. I called it "petx" and it resides in a ".pet.sfs" file. The details are in the forum at http://www.murga-linux.com/puppy/viewtopic.php?p=1038775#1038775.

s243a commented 4 years ago

I didn't see this thread when it was originally posted, but now that you re-opened it, here is my 2 cents.

In FD, we moved away from PET because of the following reasons (very similar to the points mentioned above):
* the embedding of md5sum make the tarball looks "corrupt" to tar, so it cannot be processed properly without "special tool" (basically cut out the md5sum before feeding it to tar)

* the inclusion of the "package name" as the "directory" is unnecessary and complicates matters - what if the package is renamed? What does it package manager has to do if the directory-name inside the tarball doesn't match the tarball package name?

* the petspecs is not suitable for longer package description names since everything is embedded in one long line.
My conclusion was: a) md5sum could be stored and checked in an external file, so the package becomes a normal tarball

I agree that it could be an external file and if it is an external file it doesn't have to be limited to md5 checksums. Said file could have multiple checksums and other metadata and the whole file could be signed with a pgp signature.

I wonder if you could have said external file combined in a tar with the associated package. For example .pet.tar would be a .pet file concatenated together with the above mentioned separate file containing the md5 checksum and possibly other metadata.

b) there is no need for "package directory", everything is based on "/" so that the package can be installed just by doing "tar -xf package.tar -C /". Package creation is just "tar -cf".

You might be able to do something like this

tar -x -f package.tar --keep-directory-symlink --transform='s%^package-name/%%' -T <<<package-name  -C /

So that we can extract a subdirectory as it was the root directory and avoid overwriting symbolic links.

See: https://www.gnu.org/software/tar/manual/html_section/tar_51.html https://www.gnu.org/software/tar/manual/html_node/files.html

Or you could mount the tar archive and use the pass-through options (-pdu) of cpio.

note gyrog seems to be leading this effort so I have to investigate his/her proposal.

gyrog commented 4 years ago

I wouldn't say I'm "leading this effort", I'm just the one who is stupid enough to backup my idea with "proof of concept" working Puppy code.

puppylinux-woof-CE / woof-CE

How feasible is to create an alternate pet archive format and make it the default one? #1494

Proposal

1494 (comment)