extract_firmware broken on Ubuntu 11.04 32-bit (works on 64-bit)

[GoogleCodeExporter]

What steps will reproduce the problem?
0. Use 32-bit Ubuntu 11.04 (64-bit does work)
1. sudo apt-get install -y build-essential
2. svn checkout http://firmware-mod-kit.googlecode.com/svn/trunk/ 
firmware-mod-kit-read-only
3. cd firmware-mod-kit-read-only/trunk
4. ./extract_firmware.sh ~/Download/tomato-mipsr2.trx ~/custom-tomato/

What is the expected output? What do you see instead?

 Firmware Mod Kit (extract) v0.71 beta, (c)2010 Jeremy Collake
 http://www.bitsum.com
 Checking for updates ...
  You have the latest version of this kit.
 LINUX system detected. Compatibility ok.
 Testing file system of /home/coolaj86/custom-tomato/ ...
 Building tools ...
 ERROR - Build did not succeed. Check log file, you are probably
         missing one or more necessary pre-requisites. See docs for
         a list of pre-requsites.
 CONTINUING anyway, attempting to use pre-built binaries (x32 linux)
 Preparing working directory ...
 Removing any previous files ...
 Creating directories ...
 Extracting firmware
 Attempting squashfs 3.0 lzma ...
 Trying 'damn small' variant - used by DD-WRT v24 ...
 Error: filesystem not extracted properly.
  firmware image format not compatible?

coolaj86@macbuntuair:~/Code/firmware-mod-kit-read-only/trunk$ cat extract.log 
make: Entering directory 
`/home/coolaj86/Code/firmware-mod-kit-read-only/trunk/src'
(cd ./binwalk*/src && ./configure --disable-updates && make && cp binwalk 
../../)
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking arpa/inet.h usability... yes
checking arpa/inet.h presence... no
configure: WARNING: arpa/inet.h: accepted by the compiler, rejected by the 
preprocessor!
configure: WARNING: arpa/inet.h: proceeding with the compiler's result
checking for arpa/inet.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... no
configure: WARNING: fcntl.h: accepted by the compiler, rejected by the 
preprocessor!
configure: WARNING: fcntl.h: proceeding with the compiler's result
checking for fcntl.h... yes
checking stdlib.h usability... yes
checking stdlib.h presence... no
configure: WARNING: stdlib.h: accepted by the compiler, rejected by the 
preprocessor!
configure: WARNING: stdlib.h: proceeding with the compiler's result
checking for stdlib.h... yes
checking string.h usability... yes
checking string.h presence... no
configure: WARNING: string.h: accepted by the compiler, rejected by the 
preprocessor!
configure: WARNING: string.h: proceeding with the compiler's result
checking for string.h... yes
checking unistd.h usability... yes
checking unistd.h presence... no
configure: WARNING: unistd.h: accepted by the compiler, rejected by the 
preprocessor!
configure: WARNING: unistd.h: proceeding with the compiler's result
checking for unistd.h... yes
checking for size_t... yes
checking for stdlib.h... (cached) yes
checking for GNU libc compatible malloc... no
checking for memset... no
configure: error: "Missing necessary function"
make: *** [binwalk] Error 1
make: Leaving directory 
`/home/coolaj86/Code/firmware-mod-kit-read-only/trunk/src'
 untrx 0.54 beta - (c)2006-2010 Jeremy Collake
 Opening ./tomato-K26USB-1.28.9054MIPSR2-beta-Ext.trx
 read 6602752 bytes
  Writing /home/coolaj86/custom-tomato//image_parts/segment1
    size 2672 from offset 28 ...
  Writing /home/coolaj86/custom-tomato//image_parts/segment2
    size 985460 from offset 2700 ...
 SQUASHFS magic: 0x73717368
 SQUASHFS version: 3.0
  SQUASHFS v3.0 image detected
  Writing /home/coolaj86/custom-tomato//image_parts/squashfs_magic
  Writing /home/coolaj86/custom-tomato//image_parts/squashfs-lzma-image-3_0
    size 5614592 from offset 988160 ...
  Done!

What version of the product are you using? On what operating system?

trunk

Ubuntu 11.04 32-bit

Please provide any additional information below.

Original issue reported on code.google.com by coola...@gmail.com on 27 Sep 2011 at 12:07

[GoogleCodeExporter]

Actually, it's broken on 64-bit as well. But it was working... I think it may 
have been a commit within the last week or two that broke it. I'll post back 
when I figure it out.

Original comment by coola...@gmail.com on 27 Sep 2011 at 8:01

[GoogleCodeExporter]

Looks like it's breaking during the binwalk build. This is suspect:

checking for GNU libc compatible malloc... no
checking for memset... no

It builds fine in Ubuntu 10.04, installing 11.04 now.

Original comment by heffne...@gmail.com on 27 Sep 2011 at 9:20

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I had no problems under Ubuntu 11.04 32-bit.

1. All dependencies built fine
2. Extraction of Tomato images went fine.

--> Check the build log, see where the failure was <-- 

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 9:43

[GoogleCodeExporter]

I also tested the new extract-ng.sh, it worked FINE - so both new and legacy 
(extract_firmware.sh) work in Ubuntu 11.04 32-bit .. it must be a missing 
dependency on your PC, if I had to guess. See build.log, as suggested.

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 9:46

[GoogleCodeExporter]

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 9:47

• Changed state: Invalid

[GoogleCodeExporter]

Yep, just tested it under a brand new & updated install of 11.04 (32 bit) and 
it worked fine for me too:

$ sudo apt-get install build-essential zlibg1-dev
$ cd src && make

Original comment by heffne...@gmail.com on 27 Sep 2011 at 10:05

[GoogleCodeExporter]

I don't get as far as build.log. I only have extract.log.

Both my friend and I have done fresh installs of both Ubuntu 11.04 32-bit and 
64-bit in VirtualBox on our Macs. The trimmed output of history looks akin to 
this:

    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get install -y \
      virtualbox-ose-guest-utils \
      build-essential \
      subversion \
      unrar \
      wget \
      curl \
      git \
      vim

    mkdir Code
    cd ~/Code
    svn checkout http://firmware-mod-kit.googlecode.com/svn/trunk/ firmware-mod-kit-read-only
    cd firmware-mod-kit-read-only/trunk

    curl -v 'http://downloads.sourceforge.net/project/tomatousb/Experimental%20%28beta%29/K26-MIPSR2/tomato-K26USB-1.28.9054MIPSR2-beta-Ext.rar?r=http%3A%2F%2Ftomatousb.org%2Fdownload&ts=1317161608&use_mirror=voxel' | 2>&1 | grep Location
    wget 'http://voxel.dl.sourceforge.net/project/tomatousb/Experimental%20%28beta%29/K26-MIPSR2/tomato-K26USB-1.28.9054MIPSR2-beta-Ext.rar'
    unrar x tomato-K26USB-1.28.9054MIPSR2-beta-Ext.rar
    ./extract_firmware.sh tomato-K26USB-1.28.9054MIPSR2-beta-Ext.trx ~/custom-tomato/
    # fails
    ./extract-ng.sh tomato-K26USB-1.28.9054MIPSR2-beta-Ext.trx ~/custom-tomato/
    # fails

There must be some obvious step that we're missing.

I can compile a dummy program using malloc, so that's there.

Original comment by coola...@gmail.com on 27 Sep 2011 at 10:27

[GoogleCodeExporter]

ah-ha!

    sudo apt-get install zlib1g-dev

will you please add those complete start-to-finish Ubuntu instructions to the 
wiki or give me access to do so?

Original comment by coola...@gmail.com on 27 Sep 2011 at 10:29

[GoogleCodeExporter]

Glad its working ;). I should have asked that to start with. This is the most 
common missing dependency, because very few non-typical dependencies are 
required. I will check the docs, that is about all I am good for right now, 
lol.  Heffnercj has done a wonderful job creating build/extract-ng.sh .. They 
are a gigantic leap forward ;).

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 10:51

[GoogleCodeExporter]

I don't know if you just added it now, but I do see it on the wiki.

It may be worth mentioning on http://bitsum.com/firmware_mod_kit.htm that the 
most up-to-date information is on Google Code.

And for that matter, it might be worth moving to github where it's possible to 
host the documentation along with the project in source form (markdown, etc) 
and also much stronger community involvement.

Original comment by coola...@gmail.com on 27 Sep 2011 at 10:55

[GoogleCodeExporter]

Funny you mention that, I was about to enter a new 'issue' for me to do a 301 
redirect to our new (open) docs... I am also removing the link to my page, 
putting your name above mine (alphabetical order), and adding that dependency. 
We can slowly deprecate these docs, but they can be there until the new one is 
fully done (if not already?)

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 11:08

[GoogleCodeExporter]

Uploaded to bitsum.com .. will actually proof-read that quick job later, when I 
can another min

Original comment by jeremy.collake@gmail.com on 27 Sep 2011 at 11:19

[GoogleCodeExporter]

First, sorry to keep committing, THEN proof-reading .. sometimes I should not 
be committing, but I'm moving fast - lots of projects at once. It should be 
good now, if out-dated (by my standards today, not that I am web coder).

Anyway, I propose we keep the project here, if you are satisfied I'll move the 
OLD docs to a neutral location and keep them updated on the server. On the off 
chance anyone needs the original FMK, scripts and tools, they can refer to it 
... until it is integrated as a 'chapter' of the Wiki docs here at Google.

That's my recommendation, take it for what you will.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 12:13

[GoogleCodeExporter]

http://bitsum.com/firmware_mod_kit.htm still updated again with the newer, old 
docs ... If you are intent on git, that's fine, but my reasoning is that I like 
the full history of the project to be maintained, and there is not a compelling 
enough reason to move it. Also, I'm not a git guy ;p. However, since you are 
the one working on it right now, it is YOUR call.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 12:17

[GoogleCodeExporter]

I'm setting it up on github right now. I've completed the SVN import and I'm 
setting up some of the metadata.

Original comment by coola...@gmail.com on 28 Sep 2011 at 12:40

[GoogleCodeExporter]

Ok, great ;)

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:05

[GoogleCodeExporter]

Please do add me in too, of course ;). If anyone ever comes in, sometimes after 
you've moved on, and does a rendition 3 (though I can't imagine it getting any 
better), I hope you give them the same freedom. I strongly believe in this 
concept. Sometimes F/oSS projects get didctated by people who hardly work on 
them anymore. I would humbly request you add me to the new git site though ;).

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:10

[GoogleCodeExporter]

I meant 'the same support'.. as of course they have the freedom. This prevents 
forks, ya know. I mean, if I had been like 'NO, we must keep it here', then 
you'd have to fork it. Better for me to support you. This may happen to you 
someday, if this project lives many more years ;). I'm just glad it exists. It 
was created due to the inability to build DD-WRT from source. I thought 'why do 
I need to rebuild the whole thing, I just need to change some files?', and 
wrote up this. Then, I added new firmwares as I played with other devices. I 
just hacked them in though, I didn't do it 'right' like you.

You deserve all the credit in the world for making this a totally new project.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:13

[GoogleCodeExporter]

Hope that came out right... If you've not met the type of person I am speaking 
of, you don't want to meet them, lol. Some literally will try to dictate a 
project 10 years after they quit working on it, just because they started it.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:15

[GoogleCodeExporter]

<sarcasm> And this is a chat room, right? </sarcasm>

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:17

[GoogleCodeExporter]

Last thing, please don't delete this one, I'll redirect it to git, but that way 
the change log can be preserved that way... and it will be here if ever needed 
for any crazy reason (you never know)

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:20

[GoogleCodeExporter]

I'm not Craig, I'm AJ.

I knew that most of your comments were directed at Craig, but I chimed in and 
you thought my comment was from him.

So maybe he's not even a git guy, but I know github is super easy for 
collaborators and I will probably be writing a few articles / documentation 
over the next few weeks for a project I'm working on.

I started Issue #33 for further discussion of the git / github pages stuff.

Original comment by coola...@gmail.com on 28 Sep 2011 at 1:33

[GoogleCodeExporter]

In that case, since YOU decided to move to git, I am not sure that's 
appropriate. Craig did a near rewrite.. If he wants to, that's fine, but I 
thought you were him ;o. I am an idiot, or something.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:35

[GoogleCodeExporter]

If you get my drift (those who do most should be the ones who decide)

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:36

[GoogleCodeExporter]

I'm sorry, this wouldn't be awkward, had I paid attention .. I just assumed it 
was Craig talking to me about moving the project ;o

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:37

[GoogleCodeExporter]

BUT you are legally entitled, of course, but I am not moving myself.. Craig can 
if he wants to. I am not into keeping up with what is 'cool'.. if I started a 
NEW project, I would consider it. Moving an OLD project to the latest fad seems 
inappropriate

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:39

[GoogleCodeExporter]

and I ask you to respect the original developer some, clearly I'm more than 
eager to give up all rights to anyone who 'pays their dues'.. as Craig has ;). 
So sorry, again, that I got you confused and made this awkward. If you fork the 
project to Git, that may be confusing to people, and I highly discourage it. If 
you have contributions you want to make, we can make you a committee here.

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:45

[GoogleCodeExporter]

And this kit is more than a few scripts and other people's code, I wrote an 
extensive tool to manipulate WRT54Gv5 images, for instance... AFTER I derived 
the firmare image format and checksum algorithm. I've added other tools too, to 
facilitate. Craig has done all sorts of his tools. Anyway, I got you confused 
because that was a brave request, no offense ;o. And you should have realized 
(as you said) that I was confused... please don't fork our project, please ;). 
This is MY mistake to leak you on, you have NO offense, but please don't do 
this ;)

Original comment by jeremy.collake@gmail.com on 28 Sep 2011 at 1:50