search

kartorz / utils

Some little util applications most of which are coded by script languages, e.g. python.
1 stars 0 forks source link

Feature request #2

Closed mbusb closed 10 years ago

mbusb commented 10 years ago

How about listing of all files? I hope this require only little modification in to the existing code. The feature request I make here are already exist in 7zip. But we are trying to do it pythonic way.

Also, is it possible to check if the extraction was successful? For example, I would like to extract file named "isolinux.bin". How do I know if the script extracted successfully or not as the file "isolinux.bin" may or may not exist depends on distro. Though, it is easier to find the return status by invoking system/ subprocess command. But I need python way of getting the success.

kartorz commented 10 years ago

On 03/07/2014 09:07 AM, multibootusb wrote:

How about listing of all files? I hope this require only little modification in to the existing code. The feature request I make here are already exist in 7zip. But we are trying to do it pythonic way. Can you make this clear? You mean listing of all files exinclude dirs.

Also, is it possible to check if the extraction was successful? For example, I would like to extract file named "isolinux.bin". How do I know if the script extracted successfully or not as the file "isolinux.bin" may or may not exist depends on distro. Though, it is easier to find the return status by invoking system/ subprocess command. But I need python way of getting the success. I can improve this script adding return status.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2

Regards

mbusb commented 10 years ago

"Can you make this clear? You mean listing of all files exinclude dirs."

I have made this request earlier too. Here is the example which I taken from iso9660.py and i would like to see the same from your script (though the CASE does not matter in your script:-)):-

[sundar@localhost multibootusb]$ ./iso9660.py aTinyCore-4.3.iso / /BOOT /BOOT/CORE.GZ /BOOT/ISOLINUX /BOOT/ISOLINUX/BOOT.CAT /BOOT/ISOLINUX/BOOT.MSG /BOOT/ISOLINUX/F2. /BOOT/ISOLINUX/F3. /BOOT/ISOLINUX/F4. /BOOT/ISOLINUX/ISOLINUX.BIN /BOOT/ISOLINUX/ISOLINUX.CFG /BOOT/VMLINUZ. /CDE /CDE/COPY2FS.LST /CDE/ONBOOT.LST /CDE/OPTIONAL /CDE/OPTIONAL/FLTK_1_10.TCZ /CDE/OPTIONAL/FLTK_1_10_TCZ_MD5.TXT /CDE/OPTIONAL/FLWM_TOPSIDE.TCZ /CDE/OPTIONAL/FLWM_TOPSIDE_TCZ_MD5.TXT /CDE/OPTIONAL/WBAR.TCZ /CDE/OPTIONAL/WBAR_TCZ_MD5.TXT /CDE/OPTIONAL/XLIBS.TCZ /CDE/OPTIONAL/XLIBS_TCZ_MD5.TXT /CDE/OPTIONAL/XPROGS.TCZ /CDE/OPTIONAL/XPROGS_TCZ_MD5.TXT /CDE/OPTIONAL/XVESA.TCZ /CDE/OPTIONAL/XVESA_TCZ_MD5.TXT /CDE/XBASE.LST

kartorz commented 10 years ago

I got it. This script have done yet. If don't specify "-o", it will list dirs and files, or extract files to "-o output"

./isodump.py iso:/ -r RRIP.iso

/RRIP: rrip_offset 0// //dump_dir(/)// //dump_dir (17, 800)// //==========================================// // ///boot// ///boot/grub// ///boot/grub/grub.cfg// ///linix.img// /

On 03/07/2014 09:18 AM, multibootusb wrote:

"Can you make this clear? You mean listing of all files exinclude g1dirs."

I have made this request earlier too. Here is the example which I taken from iso9660.py and i would like to see the same from your script (though the CASE does not matter in your script:-)):-

[sundar@localhost multibootusb]$ ./iso9660.py aTinyCore-4.3.iso / /BOOT /BOOT/CORE.GZ /BOOT/ISOLINUX /BOOT/ISOLINUX/BOOT.CAT /BOOT/ISOLINUX/BOOT.MSG /BOOT/ISOLINUX/F2. /BOOT/ISOLINUX/F3. /BOOT/ISOLINUX/F4. /BOOT/ISOLINUX/ISOLINUX.BIN /BOOT/ISOLINUX/ISOLINUX.CFG /BOOT/VMLINUZ. /CDE /CDE/COPY2FS.LST /CDE/ONBOOT.LST /CDE/OPTIONAL /CDE/OPTIONAL/FLTK_1_10.TCZ /CDE/OPTIONAL/FLTK_1_10_TCZ_MD5.TXT /CDE/OPTIONAL/FLWM_TOPSIDE.TCZ /CDE/OPTIONAL/FLWM_TOPSIDE_TCZ_MD5.TXT /CDE/OPTIONAL/WBAR.TCZ /CDE/OPTIONAL/WBAR_TCZ_MD5.TXT /CDE/OPTIONAL/XLIBS.TCZ /CDE/OPTIONAL/XLIBS_TCZ_MD5.TXT /CDE/OPTIONAL/XPROGS.TCZ /CDE/OPTIONAL/XPROGS_TCZ_MD5.TXT /CDE/OPTIONAL/XVESA.TCZ /CDE/OPTIONAL/XVESA_TCZ_MD5.TXT /CDE/XBASE.LST

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-36958381

Regards

mbusb commented 10 years ago

Thank you. I did not know about that. :-)

kartorz commented 10 years ago

You are welcom, Wish it help. ^_^

On 03/07/2014 09:50 AM, multibootusb wrote:

Thank you. I did not know about that. :-)

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-36960174

Regards

mbusb commented 10 years ago

I think we should be able to store the list to variable as well. I added file_list to return list of file. So that later on when isodump is called from other script, list of file can be stored in a variable. def dump_dir_r(dir_path, dire, r): file_list = [] if (dire.f_flag & 0x02) != 0x02: return dirs = read_dirs(dire.loc_extent, dire.len_data) for d in dirs: if not d.f_identifier in [".", ".."]: p = dir_path + "/" + d.f_identifier print p file_list.append(p) if r: dump_dir_r(p, d, r) return file_list

Then called the list function from other script:-

! /usr/bin/python

import isodump iso_file = "aTinyCore-4.3.iso" out_dir = "test" content = isodump.extract_directory("iso:/", "", iso_file, "") print content

However, it prints None. I think it require more modification in your script.

kartorz commented 10 years ago

On 03/08/2014 10:18 AM, multibootusb wrote:

I think we should be able to store the list to variable as well. I added file_list to return list of file. So that later on when isodump is called from other script, list of file can be stored in a variable. def dump_dir_r(dir_path, dire, r): file_list = [] if (dire.f_flag & 0x02) != 0x02: return dirs = read_dirs(dire.loc_extent, dire.len_data) for d in dirs: if not d.f_identifier in [".", ".."]: p = dir_path + "/" + d.f_identifier print p file_list.append(p) if r: dump_dir_r(p, d, r) return file_list

Then called the list function from other script:-

! /usr/bin/python

import isodump iso_file = "aTinyCore-4.3.iso" out_dir = "test" content = isodump.extract_directory("iso:/", "", iso_file, "") print content

However, it prints None. I think it require more modification in your script. It's a good idea for returning a file list. I will do it.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37086288

Regards

mbusb commented 10 years ago

I think the latest commit is broken. Tried isodump.extract_directory("iso:/", iso_cfg_ext_dir, iso_path, ".cfg"). But it does not seems to extract. The same is executed with success if run from terminal with this command "./isodump.py iso:/ -r -o ./test/ -p ".cfg" ./aTinyCore-4.3.iso"

kartorz commented 10 years ago

On 03/09/2014 10:12 AM, multibootusb wrote:

I think up the latest commit is broken. Tried isodump.extract_directory("iso:/", iso_cfg_ext_dir, iso_path, ".cfg"). But it does not seems to extract. The last commit is for making devie file. And I tested, it did work.

 Here is my test:
     isodump.extract_directory("iso:/", "tmp",

"NetBSD-6.1.1-i386.iso", ".bin")

The same is executed with success if run from terminal with this command "./isodump.py iso:/ -r -o ./test/ -p ".cfg" ./aTinyCore-4.3.iso" If this command did work, could you try isodump.extract_directory("iso:/", test, "aTinyCore-4.3.iso ", ".cfg") Maybe there are null valus among the input varialbes.

 I will commit a patch  "return" insteadof "sys.exit"  the

result(error code). Could you print the the error code?

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37116800

Regards

kartorz commented 10 years ago

On 03/08/2014 10:18 AM, multibootusb wrote:

I think we should be able to store the list to variable as well. I added file_list to return list of file. So that later on when isodump is called from other script, list of file can be stored in a variable. def dump_dir_r(dir_path, dire, r): file_list = [] if (dire.f_flag & 0x02) != 0x02: return dirs = read_dirs(dire.loc_extent, dire.len_data) for d in dirs: if not d.f_identifier in [".", ".."]: p = dir_path + "/" + d.f_identifier print p file_list.append(p) if r: dump_dir_r(p, d, r) return file_list

Then called the list function from other script:-

! /usr/bin/python

import isodump iso_file = "aTinyCore-4.3.iso" out_dir = "test" content = isodump.extract_directory("iso:/", "", iso_file, "") print content

However, it prints None. I think it require more modification in your script. Yes, It more need modification. It againsts my origin design as a little application. I should provide a library not a application. It may need a couple days.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37086288

Regards

mbusb commented 10 years ago

"It againsts my origin design as a little application." "I should provide a library not a application." I think you can keep your original goal and may just extend the functionality to satisfy developers need. Have a look at https://github.com/wummel/patool which does the same thing. However, it is heavily depended on third party tools. I assure you that there is only one script/ module in the python world to list/extract iso container and that is your's. Again, I am using your script (commit 6) without an issue. The only left over from the feature request are:- exit status iso integrity check return of file list (but still managed to work with modification in my code) As you said, implementing above features may require more modification in your script. I will be releasing a new stable version of multibootusb in couple of days and your script will be used as default instead of 7zip.

kartorz commented 10 years ago

OK, I got this and will think about it.

On 03/10/2014 10:11 AM, multibootusb wrote:

"It againsts my origin design as a little application." "I should provide a library not a application." I think you can keep your original goal and may just extend the functionality to satisfy developers need. Have a look at https://github.com/wummel/patool which does the same thing. However, it is heavily depended on third party tools. I assure you that there is only one script/ module in the python world to list/extract iso container and that is your's. Again, I am using your script (commit 6) without an issue. The only left over from the feature request are:- exit status iso integrity check return of file list (but still managed to work with modification in my code) As you said, implementing above features may require more modification in your script. I will be releasing a new stable version of multibootusb in couple of days and your script will be used as default instead of 7zip.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37148312

Regards

kartorz commented 10 years ago

Hi, I commited a new modification fixing the following issues: 1) exit status. 2) return of file list.

There is more modification, the APIs has changed.
Here is the usage:

/from isodump import ISO9660// // iso9660fs = ISO9660("NetBSD-6.1.1-i386.iso")// // ret = iso9660fs.writeDir("/", "NetBSD") # extract all the files to local directory "NetBSD//"// // if ret != 0:// // print "failure"// // // filelist = iso9660fs.readDir("/")// // for f in filelist:// // print f/

One feather "iso integrity check" left, I will do it later.

On 03/10/2014 10:11 AM, multibootusb wrote:

"It againsts my origin design as a little application." "I should provide a library not a application." I think you can keep your original goal and may just extend the functionality to satisfy developers need. Have a look at https://github.com/wummel/patool which does the same thing. However, it is heavily depended on third party tools. I assure you that there is only one script/ module in the python world to list/extract iso container and that is your's. Again, I am using your script (commit 6) without an issue. The only left over from the feature request are:- exit status iso integrity check return of file list (but still managed to work with modification in my code) As you said, implementing above features may require more modification in your script. I will be releasing a new stable version of multibootusb in couple of days and your script will be used as default instead of 7zip.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37148312

Regards

mbusb commented 10 years ago

Awesome. You are too good. ;-) will report feedback after testing. How about extraction of files using pattern match?

Just waiting for your next commit. ;-))

kartorz commented 10 years ago

Awesome. You are too good. ;-) will report feedback after testing. Thanks. I hope this script helps.

How about extraction of files using pattern match?

@path -- path within iso file system.

 # @output -- what local path you want write to.
 # @pattern -- regular expression.
 # @r -- recursion flag, write the whole sub-directories or not.
 # @all_type -- which file type should be writed.
 #              False: Write regular type files only.
 #              True: Wirte all types files (regular, device file,

link, socket, etc) def writeDir(self, path, output, pattern="", r=True, all_type=False):

 use pattern:
     iso9660fs.writeDir("/", "NetBSD", "cfg")

Don't extract/search sub-directory:
     iso9660fs.writeDir("/", "NetBSD", "cfg", False)

Just waiting for your next commit. ;-))

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37613288

Regards

mbusb commented 10 years ago

Tested the latest script and API. Works well. However, while checking random iso, I noticed that if a directory contain "." then isodum misses files to extract in to it. This happened when I was testing Salix which contain kernel file inside "huge.s" directory. You may wish to check this issue. Otherwise, it is yet another awesome commit.

"Thanks. I hope this script helps." Yes it is. One of the best feature multibootusb have got :-)

kartorz commented 10 years ago

On 03/16/2014 11:43 PM, multibootusb wrote:

Tested the latest script and API. Works well. However, while checking random iso, I noticed that if a directory contain "." then isodum misses files to extract in to it. Could you give me a example. I used "mkisofs" to test and found it should not missing files. The following is my test.

//tmp/iso/// //...// ///tmp/iso/.boot// ///tmp/iso/.boot/.boot// ///tmp/iso/boot// ///tmp/iso/boot/grub// ///tmp/iso/boot/grub/grub.cfg// ///tmp/iso/boot.s// ///tmp/iso/boot.s/boot.s/ ..

It can read and extract directory ".boot" and "boot.s".

This happened when I was testing Salix which contain kernel file inside "huge.s" directory. You may wish to check this issue. Otherwise, it is yet another awesome commit.

"Thanks. I hope this script helps." Yes it is. One of the best feature multibootusb have got :-)

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37760377

Regards

mbusb commented 10 years ago

Identified the issue. It is some thing to do with writing in to non empty directory. I am trying to extract two different directories one by one. When the second line executed, your script does not extract files as the target directory is non empty. Here is the test case:-

iso_file = "/home/sundar/Downloads/distros/salix64-xfce-14.1RC1.iso" out_dir = "test" iso9660fs = ISO9660(iso_file) iso9660fs.writeDir("/", out_dir, "kernel") iso9660fs.writeDir("/", out_dir, "isolinux")

Output of the script is :- /usr/bin/python /home/sundar/multibootusb/test.py RRIP: rrip_offset 0 The target directory is not empty Process finished with exit code 0

Here is the file list checked from terminal:- [sundar@localhost multibootusb]$ ls ./test/kernels/huge.s/ bzImage config System.map.gz [sundar@localhost multibootusb]$ ls ./test/isolinux/ [sundar@localhost multibootusb]$

Now I inter changed the lines and rerun the script:-

iso9660fs.writeDir("/", out_dir, "isolinux") iso9660fs.writeDir("/", out_dir, "kernel")

And here is the file list:- [sundar@localhost multibootusb]$ ls ./test/isolinux/ efiboot.img isolinux.bin isolinux.cfg message.txt initrd.img isolinux.boot iso.sort [sundar@localhost multibootusb]$ ls ./test/kernels/huge.s/ [sundar@localhost multibootusb]$

So the bug is identified. Time to solve it. Here are my suggestions:-

  1. Overwrite the files by default or
  2. Ask for manual/ user intervention when overwriting files or
  3. Provide one more switch [-y] to make it automatic The 3rd option is already exist in 7zip and make sense.
kartorz commented 10 years ago

Hi, I added this restriction at last commit being afraid misuse this script to override files. It seems not handy. I have cancelled this restriction. So ,It will overwrite the existing files.

On 03/17/2014 10:16 PM, multibootusb wrote:

Identified the issue. It is some thing to do with writing in to non empty directory. I am trying to extract two different directories one by one. When the second line executed, your script does not extract files as the target directory is non empty. Here is the test case:-

iso_file = "/home/sundar/Downloads/distros/salix64-xfce-14.1RC1.iso" out_dir = "test" iso9660fs = ISO9660(iso_file) iso9660fs.writeDir("/", out_dir, "kernel") iso9660fs.writeDir("/", out_dir, "isolinux")

Output of the script is :- /usr/bin/python /home/sundar/multibootusb/test.py RRIP: rrip_offset 0 The target directory is not empty Process finished with exit code 0

Here is the file list checked from terminal:- [sundar@localhost multibootusb]$ ls ./test/kernels/huge.s/ bzImage config System.map.gz [sundar@localhost multibootusb]$ ls ./test/isolinux/ [sundar@localhost multibootusb]$

Now I inter changed the lines and rerun the script:-

iso9660fs.writeDir("/", out_dir, "isolinux") iso9660fs.writeDir("/", out_dir, "kernel")

And here is the file list:- [sundar@localhost multibootusb]$ ls ./test/isolinux/ efiboot.img isolinux.bin isolinux.cfg message.txt initrd.img isolinux.boot iso.sort [sundar@localhost multibootusb]$ ls ./test/kernels/huge.s/ [sundar@localhost multibootusb]$

So the bug is identified. Time to solve it. Here are my suggestions:-

  1. Overwrite the files by default or
  2. Ask for manual/ user intervention when overwriting files or
  3. Provide one more switch [-y] to make it automatic The 3rd option is already exist in 7zip and make sense.

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-37819776

Regards

mbusb commented 10 years ago

Thank you. I have already integrated the script in to multibootusb. Now it is time for release. However, just waiting for the last feature (checking ISO ) to be implemented. So that 7zip is completely removed from dependency.

kartorz commented 10 years ago

Hi, I have finished the feature checking integrity of iso. Here is the usage: / // iso9660fs = ISO9660("xx.iso")// // iso9660fs.checkIntegrity() / the iso standrard have no like checksum ,total size stuff. What I do is checking primary volume and last file.

Regards

mbusb commented 10 years ago

OK. Will test and get back soon. I am not sure how they check the integrity of an ISO. https://fedorahosted.org/releases/i/s/isomd5sum/isomd5sum-1.0.10.tar.bz2 On Mar 19, 2014 4:36 PM, "Joni Lee" notifications@github.com wrote:

Hi, I have finished the feature checking integrity of iso. Here is the usage: / // iso9660fs = ISO9660("xx.iso")// // iso9660fs.checkIntegrity() / the iso standrard have no like checksum ,total size stuff. What I do is checking primary volume and last file.

Regards

Reply to this email directly or view it on GitHubhttps://github.com/kartorz/utils/issues/2#issuecomment-38038995 .

kartorz commented 10 years ago

I tested using "head" to cut a iso, this method should work. It can't verify error, just check capacity. Any suggestion I will appreciate.

On 03/19/2014 08:36 PM, multibootusb wrote:

OK. Will test and get back soon. I am not sure how they check the integrity of an ISO. https://fedorahosted.org/releases/i/s/isomd5sum/isomd5sum-1.0.10.tar.bz2 On Mar 19, 2014 4:36 PM, "Joni Lee" notifications@github.com wrote:

Hi, I have finished the feature checking integrity of iso. Here is the usage: / // iso9660fs = ISO9660("xx.iso")// // iso9660fs.checkIntegrity() / the iso standrard have no like checksum ,total size stuff. What I do is checking primary volume and last file.

Regards

Reply to this email directly or view it on GitHubhttps://github.com/kartorz/utils/issues/2#issuecomment-38038995 .

Reply to this email directly or view it on GitHub: https://github.com/kartorz/utils/issues/2#issuecomment-38045285

Regards

mbusb commented 10 years ago

Tested the script. There is an issue it seems with the current implementation of the checkIntegrity() function. I had a corrupted ISO which I subjected to test with 7zip and isodump.py. 7zip correctly identified as corrupted and yours not. Here is the output of terminal when tested using 7zip:- Testing KNOPPIX Testing KNOPPIX/KNOPPIX Testing KNOPPIX/KNOPPIX.tmp ERROR: E_FAIL
That means, in 7zip, each file of an ISO is subjected to integrity test.

mbusb commented 10 years ago

Here is another test case using three ISOs:- [sundar@localhost multibootusb]$ isovfy aTinyCore-4.3.iso Root at extent 1d, 2048 bytes [0 0] No errors found [sundar@localhost multibootusb]$ isovfy ../CentOS-6.5-x86_64-bin-DVD1.iso isovfy: Short read on old image [sundar@localhost multibootusb]$ isovfy /media/Fun/distros/ADRIANE_KNOPPIX_V7.0.4CD-2012-08-20-EN.iso Root at extent 1d, 2048 bytes [0 0] No errors found [sundar@localhost multibootusb]$ ADRIANEKNOPPIX is the ISO I was talking about in the earlier post. even isovfy could not check the interity of an ISO.