firmware not extracting correctly

[GoogleCodeExporter]

What steps will reproduce the problem?
1. ./extract_firmware.sh CUSTOM.BIN.SAVED_20120505_110558 ../BTHomeHub2A_8.1.H.U
2. ./extract-ng.sh CUSTOM.BIN.SAVED_20120505_110558 ../BTHomeHub2A_8.1.H.U

What is the expected output? What do you see instead?

i expected files to be extracted in ../BTHomeHub2A_8.1.H.U/

1. Error: filesystem not extracted properly.
  firmware image format not compatible?

2. Scanning firmware...

DECIMAL     HEX         DESCRIPTION
--------------------------------------------------------------------------------
-----------------------
2322432     0x237000    Squashfs filesystem, big endian, lzma signature, version 
3.1, size: 1948265 bytes, 55 inodes, blocksize: 65536 bytes, created: Fri Apr 
15 06:30:53 2011
4325376     0x420000    Squashfs filesystem, big endian, lzma signature, version 
3.1, size: 6170550 bytes, 473 inodes, blocksize: 65536 bytes, created: Fri Apr 
15 06:27:45 2011
10620928    0xA21000    Squashfs filesystem, big endian, lzma signature, version 
3.1, size: 247 bytes, 4 inodes, blocksize: 65536 bytes, created: Fri Apr 15 
06:26:53 2011

Extracting 10620928 bytes of  header image at offset 0
Extracting squashfs file system at offset 10620928
Extracting squashfs files...
Firmware extraction successful!
Firmware parts can be found in '../BTHomeHub2A_8.1.H.U/*'

What version of the product are you using? On what operating system?
$ svn update
At revision 309.
$ uname -a
Linux ubuntu 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:10:02 UTC 2010 i686 
GNU/Linux

Please provide any additional information below.

$ find ../BTHomeHub2A_8.1.H.U/ -type f
../BTHomeHub2A_8.1.H.U/logs/binwalk.log
../BTHomeHub2A_8.1.H.U/logs/config.log
../BTHomeHub2A_8.1.H.U/image_parts/header.img
../BTHomeHub2A_8.1.H.U/image_parts/rootfs.img

$ ls ../BTHomeHub2A_8.1.H.U/image_parts/ -lha
total 17M
drwxr-xr-x 2 ubuntu ubuntu 4.0K 2012-05-08 19:34 .
drwxr-xr-x 5 ubuntu ubuntu 4.0K 2012-05-08 19:37 ..
-rw-r--r-- 1 ubuntu ubuntu  11M 2012-05-08 19:34 header.img
-rw-r--r-- 1 ubuntu ubuntu 5.9M 2012-05-08 19:34 rootfs.img

$ ls ../CUSTOM.BIN.SAVED_20120505_110558 -lha
-rw-r--r-- 1 ubuntu ubuntu 16M 2012-05-05 12:25 
../CUSTOM.BIN.SAVED_20120505_110558

$ find ../BTHomeHub2A_8.1.H.U/ -type d
../BTHomeHub2A_8.1.H.U/
../BTHomeHub2A_8.1.H.U/logs
../BTHomeHub2A_8.1.H.U/image_parts
../BTHomeHub2A_8.1.H.U/rootfs
../BTHomeHub2A_8.1.H.U/rootfs/sbin
../BTHomeHub2A_8.1.H.U/rootfs/usr
../BTHomeHub2A_8.1.H.U/rootfs/usr/bin

$ cat ../BTHomeHub2A_8.1.H.U/logs/config.log 
FW_SIZE='16777216'
HEADER_TYPE=''
HEADER_SIZE=''
HEADER_IMAGE_SIZE='10620928'
HEADER_IMAGE_OFFSET='0'
FOOTER_SIZE='0'
FOOTER_OFFSET='16777216'
FS_TYPE='squashfs'
FS_OFFSET='10620928'
ENDIANESS='-be'
MKFS="./src/others/squashfs-3.3/mksquashfs"

i got the following from the serial console, hope it helps.

squashfs: version 3.3 (2007/10/31) Phillip Lougher
squashfs: LZMA suppport for slax.org by jro

parse_bthub_partitions (block_size: 131072)
bthub: header of CORE partition is detected
bthub: kernel is at offset (be1a001a), size (96a15)
corefs_pattern: info is at offset (bea03004), size (4) including a header of (4)
corefs_pattern: ds is at offset (bea03008), size (24) including a header of (4)
bthub: a signature has been stored
bthub: SquashFS is expected at be237000, found magic is 71736873
bthub: SquashFS (start: be237000, size 1e9000)
bthub: SquashFS is expected at be420000, found magic is 71736873
bthub: SquashFS (start: be420000, size 600000)
bthub: user partition is created
bthub: header of EXTENDED partitin is detected
extended_pattern: info is at offset (bea22004), size (4) including a header of 
(4)
extended_pattern: ds is at offset (bea22008), size (24) including a header of 
(4)
bthub: a signature has been stored
bthub: SquashFS is expected at bea21000, found magic is 71736873
bthub: SquashFS (start: bea21000, size 1f000)
bthub mtd: user (be020000, 180000), write(1), sdram(0)
bthub mtd: rootfs (be420000, 600000), write(0), sdram(0)
bthub mtd: modfs (be237000, 1e9000), write(0), sdram(0)
bthub mtd: extfs (bea21000, 1f000), write(0), sdram(0)
bthub mtd: extended (bea20000, 5e0000), write(1), sdram(0)
bthub mtd: flash (be000000, 1000000), write(1), sdram(0)

6 BTHub partitions found on MTD device flash

Creating 6 MTD partitions on "flash":
0x00020000-0x001a0000 : "user"
0x00420000-0x00a20000 : "rootfs"
0x00237000-0x00420000 : "modfs"
0x00a21000-0x00a40000 : "extfs"
0x00a20000-0x01000000 : "extended"
0x00000000-0x01000000 : "flash"

Original issue reported on code.google.com by jf1...@gmail.com on 9 May 2012 at 1:46

Attachments:

• BTHomeHub2A_Boot.log

[GoogleCodeExporter]

The build-ng script does not currently support firmware that contains multiple 
file systems. In this case, it extracted the last detected file system, but 
based on the inode count, the primary file system appears to be the second one 
at offset 0x420000.

I will add an option to allow the user to select which file system they want to 
extract.

Original comment by heffne...@gmail.com on 8 Jul 2012 at 12:11

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

So i've been having a dig about today and found the following

http://download.modem-help.co.uk/mfcs-B/BT/Home-Hub-2/A/Non-BT-Network-Usage/Fla
shWithoutJTAG-btsimonh-v1-1.7z.php?showFile=

README
{
# Author David Hone
# Applications unpacks and packs Thomson Modem binaries, these binaries are in 
form  
# of tar file with a general format [ name length ] data. 
# The first application unpackth-1 should be called as follows "unpackth-1 -d 
# ZY01AA8.1HG" 
.....
to rebuild the packed image as follows ./packth-1 ZY01AA8.1HG.lst 
This will build a new packed image called "NewCompressedFile". All you then 
need to do is rename it back to ZY01AA8.1HG when ready to deploy. 
}

Directory: FlashWithoutJTAG_btsimonh_v1.1/x86KLinuxTools
btv2squashfs.zip
thomunpack.zip

Original comment by jf1...@gmail.com on 9 Oct 2012 at 1:05

[GoogleCodeExporter]

unpackth-1.c

// Author David this application is free too distrubute around the linux 
commuitity 

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
//#include <iostream.h>
#include <string.h>

struct unPackFile {
char *fileName;
char *sizePtr;
char *endPtr;
int fileSize;
char *fileData;

}_unPackFile;

int onlyDisplayGZFiles=0;
int decompFile=0;

int createDirectory(char *filename)
{
// Cretae the required directory
    int cnt=0,status=0;
    char *dirptr;
    dirptr = calloc(strlen(filename)+1,sizeof(char));
    strcpy(dirptr,filename);
    if (dirptr == NULL)
    {
       printf("Failed to create directory buffer\n");
       exit(1);
    }

    while (dirptr[cnt] != 0x00)
    {
      if (dirptr[cnt] == '/')
      {
         dirptr[cnt]=0;
         status = mkdir(dirptr,0777);
         dirptr[cnt]='/';

      }
      cnt++;
    }
   free(dirptr);
}

int CheckforOption(char *item)
{
    int cnt=0;
        int optset=0;
        char opttype=0;
    while (item[cnt] !=0)
        {
            if (optset == 1)
            {
//            printf("Found option marker\n");
            switch (toupper(item[cnt]))
              {
              case 'G' : onlyDisplayGZFiles=1;
                         break;
              case 'D' : decompFile=1;
                        break;
              default : break;

              }           
            optset=0; 
        }
            else 
            {
        if (item[cnt] == '-')
                   optset=1;
            }
        cnt++;
        }

}

int main(int argc,char *argv[])
{
  int i,c,t;
  unsigned short swap1,swap2;
  char testChar;
  FILE *fpw,*fpr,*fpl;
  if (argc < 2) 
  {
    printf("Requires input file\n");
    exit(1);

  } 
  char *fileToProcess;   
  int op=0;
  for (op=1; op < argc ; op++)
  {
      CheckforOption(argv[op]);
      if (!strstr(argv[op],"-"))
         fileToProcess=argv[op];
  }

  printf("File to process %s\n",fileToProcess);
 // exit(1);
  if((fpr = fopen(fileToProcess, "rb")) == NULL) 
  { 
    printf("Cannot open file %s.\n",argv[1]); 
    exit(1);
  }
  char tempbuf[260];
  char *tmpchrptr = (char *)&tempbuf;
  sprintf(tmpchrptr,"%s.lst",fileToProcess); 
  if ((fpl = fopen(tmpchrptr,"wb")) == NULL)
  {
     printf("Unable to open list file");
  }
  i=0; 
   while(!feof(fpr))  
   {  
    memset(&_unPackFile,0x00,sizeof(_unPackFile));
    t=0; 
    testChar=fgetc(fpr);
    if (testChar == EOF)
    {
        printf("End of file found\n");
    break;
    }
    if (testChar != '[')
    {
       printf("Error expected to find '[' but found %2.2x possible end of file!!!",testChar,i);
       break;
    } 
    i++;
    testChar=fgetc(fpr);
    if (testChar != ' ')
    {
       printf("Error expected to find space at postion %d\n",i);
       exit(1);
    }
    i++;
    c=i;
    while ((testChar=fgetc(fpr)) != ']')
    {
       if (testChar == ' ')
          t++;
       i++;
    }

    // Now have the end of text length, now rewind back 
    fseek(fpr,c,SEEK_SET);
    _unPackFile.fileName=calloc((i-c)+1,sizeof(char));
    fread(_unPackFile.fileName,i-c,sizeof(char),fpr);
//    printf("Debug: %s\n",_unPackFile.fileName);

    _unPackFile.sizePtr=_unPackFile.fileName;
    _unPackFile.sizePtr++;  // Get past first blank 
    // Inc pointer along to size value
    while (*(_unPackFile.sizePtr) != ' ')
    {

        _unPackFile.sizePtr++;    

    }    
    *(_unPackFile.sizePtr++)=0; // Termainate name
    _unPackFile.endPtr=_unPackFile.sizePtr;
    // Check for binary or Ascii file, if binary we get a length value
    if (t > 1)
    {
//      printf("Binary\n");
      while (*(_unPackFile.endPtr) != ' ')
      {
       _unPackFile.endPtr++;
      }
      _unPackFile.endPtr=0; // Termianate count    
      _unPackFile.fileSize = atoi(_unPackFile.sizePtr); /* convert string to int then add it to sum */
      // Now read data content of file
      _unPackFile.fileData=calloc(_unPackFile.fileSize+1,sizeof(char));
      i+=_unPackFile.fileSize;
      testChar=fgetc(fpr); // Trash ']' 
      testChar=fgetc(fpr); // 
      if (testChar != 0x0a) 
      {
         printf("Expected started of data block, but got %2.2x",testChar);
         exit(1);
      }
      fread(_unPackFile.fileData,_unPackFile.fileSize,sizeof(char),fpr);
    }
    else
    {
//      printf("Ascii\n");
      // Move the end pointer on to the start of the asic block
      testChar=fgetc(fpr); // Trash ']' 
      testChar=fgetc(fpr);
      if (testChar != 0x0a)
      {
         printf("Expected start of ascii block, but got %2.2.x",testChar);
         exit(1);
      }      
//      printf("About to loop forward on ascii block\n");
      c=0;      
      int bc=0; 
      // Read ascii unti end of block
      char endofblock[] = {0x0a,0x0a,0x5b,0x00};      
      while (!feof(fpr))
      {

         testChar=fgetc(fpr);
         if (testChar == endofblock[bc])
         {      
           bc++;
           if (endofblock[bc] == 0x00)
           {
              break;
           }
         }
         else
            bc=0;
         c++;
      }      
      fseek(fpr,i+2,SEEK_SET);
      _unPackFile.fileData=calloc(c+1,sizeof(char));
      i=(c-1)+i;
      fread(_unPackFile.fileData,c-1,sizeof(char),fpr);
      _unPackFile.fileSize=c-1;  
//      printf("%s",_unPackFile.fileData); 
//      exit(1);
    }
    testChar=fgetc(fpr);
    if (testChar != 0x0a)
    {
       printf("Expected 0x0a at the end of the data block,we got %2.2x, we had %d data blocks\n",testChar,t);
       exit(1); 
    }
    i+=3;

    createDirectory(_unPackFile.fileName);

    if((fpw = fopen(_unPackFile.fileName, "wb")) == NULL) 
    {
        printf("Cannot open file %s.\n",_unPackFile.fileName);
        exit(1);
    }
    fwrite(_unPackFile.fileData,_unPackFile.fileSize,sizeof(char),fpw);
    fwrite(_unPackFile.fileName,strlen(_unPackFile.fileName),sizeof(char),fpl);
    fputc('\n',fpl);
    fclose(fpw);
//    printf("Name = %s, Size = %s converted value =%d, blocks 
%d\n",_unPackFile.fileName,_unPackFile.sizePtr,_unPackFile.fileSize,t);
    if (onlyDisplayGZFiles == 1)
    {
    if (strstr(_unPackFile.fileName,".gz"))
        {
          printf("%s\n",_unPackFile.fileName);
        }
    }
    else 
    {    
       printf("%s\n",_unPackFile.fileName);
    }
    if (decompFile == 1)
    {
//        printf("Decomp set\n");
        if (strstr(_unPackFile.fileName,".gz"))
        {

           FILE *fprgz;
           if ((fprgz = fopen(_unPackFile.fileName,"rb")) == NULL)
       {
                printf("Unable to open %s for sampling\n",_unPackFile.fileName);
        exit(1);

        }
        // Get gzip flags
            char gzipName[160];
            gzipName[0]=0;
        int cnt=0;
        unsigned char gzflgs;
        fseek(fprgz,3,SEEK_SET);
        gzflgs = fgetc(fprgz);
        printf("gzFlags = %2.2x\n",gzflgs); 
        if ((gzflgs & 0x08) != 0)
        { 

           fseek(fprgz,10,SEEK_SET);
           unsigned char namec;
           while ((namec = fgetc(fprgz)) != 0)
           {
                  gzipName[cnt++]=namec;
                  gzipName[cnt]=0;
           }
               printf("Gzip has file name %s\n",gzipName);
            }   
            fclose(fprgz);
        fprintf(fpl,"%s\n",gzipName);
            const char gzPrg[] = "/usr/bin/gzip";

            char * CmdLine = calloc(strlen(gzPrg) + strlen( _unPackFile.fileName) + 20,sizeof(char));
            sprintf(CmdLine,"%s -dN %s\n",gzPrg,_unPackFile.fileName);

            printf("About to execute %s",CmdLine);
            system(CmdLine);
            free(CmdLine);  
        }     
    }
    free(_unPackFile.fileName);
    free(_unPackFile.fileData);

  }
  fclose(fpr);
  fclose(fpl);
  return 0;
}

Original comment by jf1...@gmail.com on 9 Oct 2012 at 1:06

[GoogleCodeExporter]

packth-1

// Author David this application is free too distrubute around the linux 
commuitity 
// Package back up files

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
//#include <iostream.h>
#include <string.h>

struct unPackFile {
char *fileName;
char *sizePtr;
char *endPtr;
int fileSize;
char *fileData;
#define MAXPATHLEN 300
}_unPackFile;

int onlyDisplayGZFiles=0;
int decompFile=0;

int freadline  (FILE *  fp,char * string)  
{
     int     c;
     int     i = 0;

     while (isspace(c = getc(fp)));

     if (c == EOF)
         return (EOF);
     do
    {
         string[i++] = c;
    } while ((c = getc(fp)) != '\n' && c != EOF && i<MAXPATHLEN-1);
     string[i++] = 0;
     return (i);
}

int createDirectory(char *filename)
{
// Cretae the required directory
    int cnt=0,status=0;
    char *dirptr;
    dirptr = calloc(strlen(filename)+1,sizeof(char));
    strcpy(dirptr,filename);
    if (dirptr == NULL)
    {
       printf("Failed to create directory buffer\n");
       exit(1);
    }

    while (dirptr[cnt] != 0x00)
    {
      if (dirptr[cnt] == '/')
      {
         dirptr[cnt]=0;
         status = mkdir(dirptr,0777);
         dirptr[cnt]='/';

      }
      cnt++;
    }
   free(dirptr);
}

int CheckforOption(char *item)
{
    int cnt=0;
        int optset=0;
        char opttype=0;
    while (item[cnt] !=0)
        {
            if (optset == 1)
            {
//            printf("Found option marker\n");
            switch (toupper(item[cnt]))
              {
              case 'G' : onlyDisplayGZFiles=1;
                         break;
              case 'D' : decompFile=1;
                        break;
              default : break;

              }           
            optset=0; 
        }
            else 
            {
        if (item[cnt] == '-')
                   optset=1;
            }
        cnt++;
        }

}

//
// Copy file fpr to fpw
//

int CopyFileToFile(FILE *fpw,FILE *fpr)
{
      char *tmpbuff;    
      fseek(fpr,0,SEEK_END);
      int fsize = ftell(fpr);
      if (fsize)
      {
         fseek(fpr,0,SEEK_SET);
         tmpbuff = calloc(fsize+1,sizeof(char));
         fread(tmpbuff,fsize,sizeof(char),fpr);
         fwrite(tmpbuff,fsize,sizeof(char),fpw); 
         free(tmpbuff);
      }     
      return 0; 
}

int CheckAorB(char *string,FILE *fpw,FILE *fpl)
{
    FILE *fpr;
    char *tmpstr;
    char *gz;
    int binaryValue=0;
    int dottext =0;
    const char *FileExtList[] = {".txt",".types",".conf",NULL}; 
    int Fl=0;
    while (FileExtList[Fl] != NULL)
    {
       if (strstr(string,FileExtList[Fl]))
       {    
          dottext=1;
          break;
       }
       Fl++;
    }

    if (strstr(string,".gz"))
    {
     binaryValue=1;
     // Convert original file to gz
     printf("Binary route\n");
    // gzip file in here

     char *fileToGz;
     tmpstr = calloc(strlen(string)+40,sizeof(char));
     fileToGz = calloc(strlen(string)+40,sizeof(char));     

     strcpy(tmpstr,string); // duplicate string for changing
     strcpy(fileToGz,string);
     gz=strstr(tmpstr,".gz");
     if (gz == NULL)
      {
       printf("Something went wrong looking for .gz\n");
           exit(1);
      } 
     int endOfStr;
     endOfStr = strlen(fileToGz);
     while (endOfStr > 0)
     {
        if (fileToGz[endOfStr] == '/')
    {
        fileToGz[++endOfStr]=0;
            break;
    }
        endOfStr--;
     }
     freadline(fpl,fileToGz+endOfStr);

     char *spawnApp;
     const char gzCmd[]="/usr/bin/gzip";

     spawnApp =calloc(strlen(gzCmd) + strlen(tmpstr)+80,sizeof(char));
     sprintf(spawnApp,"%s -Nc %s > %s\n",gzCmd,fileToGz,tmpstr);
     printf("About to execute %s\n",spawnApp);
     system(spawnApp);
     free(spawnApp);
     remove(fileToGz);  
     free(tmpstr);
     free(fileToGz);
     }
     if ((fpr = fopen(string,"rb")) == NULL)
     {
        printf("Error unable to open %s for reading\n",string);
        exit(1);
     }

    // Now we are ready to write out compressed file data
     if (dottext == 0)
     {
        printf("Processing binary file\n");
            // We have to get the binary file size and write this size in 
            fputc(' ',fpw); // Blank before file size value
        fseek(fpr,0,SEEK_END); // File size to
            int fize = ftell(fpr);
        printf("Binary file size %d\n",fize);
        fseek(fpr,0,SEEK_SET); // Reset to start of file    
        fprintf(fpw,"%d ]%c",fize,0x0a);
        CopyFileToFile(fpw,fpr);        
     }
    else
     { // Ascii 
            printf("Processing ascii file\n");
            fprintf(fpw," ]%c",0x0a);
        CopyFileToFile(fpw,fpr);    
     }
    fputc(0x0a,fpw);  // Place end of data marker
    fclose(fpr);
    return (0); 
}

//
//  Main
//
int main(int argc,char *argv[])
{
  int i,c,t;
  unsigned short swap1,swap2;
  char testChar;
  FILE *fpw,*fpr,*fpl;
  if (argc < 2) 
  {
    printf("Requires input file\n");
    exit(1);

  } 
  char *fileToProcess;   
  int op=0;
  for (op=1; op < argc ; op++)
  {
      CheckforOption(argv[op]);
      if (!strstr(argv[op],"-"))
         fileToProcess=argv[op];
  }

  printf("File to process %s\n",fileToProcess);

  if ((fpr = fopen(fileToProcess,"rb")) == NULL)
  {
     printf("Could not open list file %s\n",fileToProcess);
     exit(1);
  }
  char outPutFile[] = "NewCompressedFile";
  if ((fpw = fopen (outPutFile,"wb")) == NULL)
  {
     printf("Failed to open %s for writing\n",outPutFile);
     exit(1);
  }
  char *chrPtrLine = (char *)calloc(MAXPATHLEN,sizeof(char));
  char *tmpchr = (char *)calloc(MAXPATHLEN,sizeof(char));
  while (!feof(fpr))
  {
    int binaryValue;  

    freadline(fpr,chrPtrLine);
    printf("Processing %s\n",chrPtrLine);   
    strcpy(tmpchr,chrPtrLine);

    // Write  start of label with trailing blank
    fputc('[',fpw);
    fputc(' ',fpw); 
    fwrite(chrPtrLine,strlen(chrPtrLine),sizeof(char),fpw);

    binaryValue = CheckAorB(tmpchr,fpw,fpr);
  }
  fclose(fpr);
  fclose(fpw);
  free(tmpchr);
  free(chrPtrLine);
  return 0;
}

Original comment by jf1...@gmail.com on 9 Oct 2012 at 1:07

[GoogleCodeExporter]

Original comment by jeremy.collake@gmail.com on 11 Jun 2013 at 8:14

• Changed state: WontFix