m0hadang
commented
2 years ago - https://www.gnu.org/software/tar/manual/html_node/Standard.html
- https://joy24.tistory.com/84?category=391670
- https://github.com/calccrypto/tar
Open m0hadang opened 2 years ago
m0hadang
commented
2 years ago m0hadang
commented
2 years ago Basic Tar Format (This message will disappear, once this node revised.)
While an archive may contain many files, the archive itself is a single ordinary file. Like any other file, an archive file can be written to a storage device such as a tape or disk, sent through a pipe or over a network, saved on the active file system, or even stored in another archive. An archive file is not easy to read or manipulate without using the tar utility or Tar mode in GNU Emacs.
Physically, an archive consists of a series of file entries terminated by an end-of-archive entry, which consists of two 512 blocks of zero bytes.
A file entry usually describes one of the files in the archive (an archive member), and consists of a file header and the contents of the file.
File headers contain file names and statistics, checksum information which tar uses to detect file corruption, and information about file types.
Archives are permitted to have more than one member with the same member name.
In addition to entries describing archive members, an archive may contain entries which tar itself uses to store information.
A tar archive file contains a series of blocks.
Each file archived is represented by a header block which describes the file, followed by zero or more blocks which give the contents of the file.
At the end of the archive file there are two 512-byte blocks filled with binary zeros as an end-of-file marker.
A reasonable system should write such end-of-file marker at the end of an archive, but must not assume that such a block exists when reading an archive.
In particular GNU tar always issues a warning if it does not encounter it.
The blocks may be blocked for physical I/O operations.
Each record of n blocks (where n is set by the ‘--blocking-factor=512-size’ (‘-b 512-size’) option to tar) is written with a single ‘write ()’ operation.
On magnetic tapes, the result of such a write is a single record.
When writing an archive, the last record of blocks should be written at the full size, with blocks after the zero block containing all zeros.
When reading an archive, a reasonable system should properly handle an archive whose last record is shorter than the rest, or which contains garbage records after a zero block.
The header block is defined in C as follows. In the GNU tar distribution, this is part of file ‘src/tar.h’:
/ tar Header Block, from POSIX 1003.1-1990. /
/ POSIX header. / - 헤더 블록 struct posix_header { / byte offset / char name[100]; / 0 / char mode[8]; / 100 / char uid[8]; / 108 / char gid[8]; / 116 / char size[12]; / 124 / char mtime[12]; / 136 / char chksum[8]; / 148 / char typeflag; / 156 / char linkname[100]; / 157 / char magic[6]; / 257 / char version[2]; / 263 / char uname[32]; / 265 / char gname[32]; / 297 / char devmajor[8]; / 329 / char devminor[8]; / 337 / char prefix[155]; / 345 / / 500 / };
/ Values used in typeflag field. / - typeflag 필드 에서 사용
next file in the archive *// Bits used in the mode field, values in octal. / - mode 필드 에서 사용
/* file permissions *// tar Header Block, GNU extensions. /
/ In GNU tar, SYMTYPE is for to symbolic links, and CONTTYPE is for contiguous files, so maybe disobeying the "reserved" comment in POSIX header description. I suspect these were meant to be used this way, and should not have really been "reserved" in the published standards. /
/ BEWARE BEWARE BEWARE that the following information is still boiling, and may change. Even if the OLDGNU format description should be accurate, the so-called GNU format is not yet fully decided. It is surely meant to use only extensions allowed by POSIX, but the sketch below repeats some ugliness from the OLDGNU format, which should rather go away. Sparse files should be saved in such a way that they do not require two passes at archive creation time. Huge files get some POSIX fields to overflow, alternate solutions have to be sought for this. /
/ Descriptor for a single file hole. /
struct sparse { / byte offset / char offset[12]; / 0 / char numbytes[12]; / 12 / / 24 / };
/ Sparse files are not supported in POSIX ustar format. For sparse files with a POSIX header, a GNU extra header is provided which holds overall sparse information and a few sparse descriptors. When an old GNU header replaces both the POSIX header and the GNU extra header, it holds some sparse descriptors too. Whether POSIX or not, if more sparse descriptors are still needed, they are put into as many successive sparse headers as necessary. The following constants tell how many sparse descriptors fit in each kind of header able to hold them. /
/ Extension header for sparse files, used immediately after the GNU extra header, and used only if all sparse information cannot fit into that extra header. There might even be many such extension headers, one after the other, until all sparse information has been recorded. /
struct sparse_header { / byte offset / struct sparse sp[SPARSES_IN_SPARSE_HEADER]; / 0 / char isextended; / 504 / / 505 / };
/ The old GNU format header conflicts with POSIX format in such a way that POSIX archives may fool old GNU tar's, and POSIX tar's might well be fooled by old GNU tar archives. An old GNU format header uses the space used by the prefix field in a POSIX header, and cumulates information normally found in a GNU extra header. With an old GNU tar header, we never see any POSIX header nor GNU extra header. Supplementary sparse headers are allowed, however. /
struct oldgnu_header { / byte offset / char unused_pad1[345]; / 0 / char atime[12]; / 345 Incr. archive: atime of the file / char ctime[12]; / 357 Incr. archive: ctime of the file / char offset[12]; / 369 Multivolume archive: the offset of the start of this volume / char longnames[4]; / 381 Not used / char unused_pad2; / 385 / struct sparse sp[SPARSES_IN_OLDGNU_HEADER]; / 386 / char isextended; / 482 Sparse file: Extension sparse header follows / char realsize[12]; / 483 Sparse file: Real size/ / 495 / };
/ OLDGNU_MAGIC uses both magic and version fields, which are contiguous. Found in an archive, it indicates an old GNU header format, which will be hopefully become obsolescent. With OLDGNU_MAGIC, uname and gname are valid, though the header is not truly POSIX conforming. /
/* The standards committee allows only capital A through capital Z for user-defined expansion. Other letters in use include:
'A' Solaris Access Control List 'E' Solaris Extended Attribute File 'I' Inode only, as in 'star' 'N' Obsolete GNU tar, for file names that do not fit into the main header. 'X' POSIX 1003.1-2001 eXtended (VU version) */
/ This is a dir entry that contains the names of files that were in the dir at the time the dump was made. /
/ Identifies the next file on the tape as having a long linkname. /
/ Identifies the next file on the tape as having a long name. /
/ This is the continuation of a file that began on another volume. /
/ This is for sparse files. /
/ This file is a tape/volume header. Ignore it on extraction. /
/ Solaris extended header /
/ Jörg Schilling star header /
struct star_header { / byte offset / char name[100]; / 0 / char mode[8]; / 100 / char uid[8]; / 108 / char gid[8]; / 116 / char size[12]; / 124 / char mtime[12]; / 136 / char chksum[8]; / 148 / char typeflag; / 156 / char linkname[100]; / 157 / char magic[6]; / 257 / char version[2]; / 263 / char uname[32]; / 265 / char gname[32]; / 297 / char devmajor[8]; / 329 / char devminor[8]; / 337 / char prefix[131]; / 345 / char atime[12]; / 476 / char ctime[12]; / 488 / / 500 / };
struct star_in_header { char fill[345]; / 0 Everything that is before t_prefix / char prefix[1]; / 345 t_name prefix / char fill2; / 346 / char fill3[8]; / 347 / char isextended; / 355 / struct sparse sp[SPARSES_IN_STAR_HEADER]; / 356 / char realsize[12]; / 452 Actual size of the file / char offset[12]; / 464 Offset of multivolume contents / char atime[12]; / 476 / char ctime[12]; / 488 / char mfill[8]; / 500 / char xmagic[4]; / 508 "tar" / };
struct star_ext_header { struct sparse sp[SPARSES_IN_STAR_EXT_HEADER]; char isextended; };
All characters in header blocks are represented by using 8-bit characters in the local variant of ASCII.
Each field within the structure is contiguous; that is, there is no padding used within the structure.
Each character on the archive medium is stored contiguously.
Bytes representing the contents of files (after the header block of each file) are not translated in any way and are not constrained to represent characters in any character set.
The tar format does not distinguish text files from binary files, and no translation of file contents is performed.
The name, linkname, magic, uname, and gname are null-terminated character strings.
All other fields are zero-filled octal numbers in ASCII. Each numeric field of width w contains w minus 1 digits, and a null. (In the extended GNU format, the numeric fields can take other forms.)
The name field is the file name of the file, with directory names (if any) preceding the file name, separated by slashes.
The mode field provides nine bits specifying file permissions and three bits to specify the Set UID, Set GID, and Save Text (sticky) modes.
Values for these bits are defined above.
When special permissions are required to create a file with a given mode, and the user restoring files from the archive does not hold such permissions, the mode bit(s) specifying those special permissions are ignored.
Modes which are not supported by the operating system restoring files from the archive will be ignored.
Unsupported modes should be faked up when creating or updating an archive; e.g., the group permission could be copied from the other permission.
The uid and gid fields are the numeric user and group ID of the file owners, respectively. If the operating system does not support numeric user or group IDs, these fields should be ignored.
The size field is the size of the file in bytes; linked files are archived with this field specified as zero.
The mtime field represents the data modification time of the file at the time it was archived. It represents the integer number of seconds since January 1, 1970, 00:00 Coordinated Universal Time.
The chksum field represents the simple sum of all bytes in the header block.
Each 8-bit byte in the header is added to an unsigned integer, initialized to zero, the precision of which shall be no less than seventeen bits. When calculating the checksum, the chksum field is treated as if it were all blanks.
The typeflag field specifies the type of file archived.
If a particular implementation does not recognize or permit the specified type, the file will be extracted as if it were a regular file.
The atime and ctime fields are used in making incremental backups; they store, respectively, the particular file’s access and status change times.
The offset is used by the ‘--multi-volume’ (‘-M’) option, when making a multi-volume archive.
The following fields were added to deal with sparse files.
This is the method tar uses to detect a sparse file, and once such a file is detected, it is treated differently from non-sparse files.
Sparse files are often dbm files, or other database-type files which have data at some points and emptiness in the greater part of the file. Such files can appear to be very large when an ‘ls -l’ is done on them, when in truth, there may be a very small amount of important data contained in the file. It is thus undesirable to have tar think that it must back up this entire file, as great quantities of room are wasted on empty blocks, which can lead to running out of room on a tape far earlier than is necessary.
Thus, sparse files are dealt with so that these empty blocks are not written to the tape. Instead, what is written to the tape is a description, of sorts, of the sparse file: where the holes are, how big the holes are, and how much data is found at the end of the hole. This way, the file takes up potentially far less room on the tape, and when the file is extracted later on, it will look exactly the way it looked beforehand. The following is a description of the fields used to handle a sparse file:
The sp is an array of struct sparse. Each struct sparse contains two 12-character strings which represent an offset into the file and a number of bytes to be written at that offset. The offset is absolute, and not relative to the offset in preceding array element.
The header can hold four of these struct sparse at the moment; if more are needed, they are not stored in the header.
The isextended flag is set when an extended_header is needed to deal with a file. Note that this means that this flag can only be set when dealing with a sparse file, and it is only set in the event that the description of the file will not fit in the allotted room for sparse structures in the header. In other words, an extended_header is needed.
The extended_header structure is used for sparse files which need more sparse structures than can fit in the header. The header can fit 4 such structures; if more are needed, the flag isextended gets set and the next block is an extended_header.
Each extended_header structure contains an array of 21 sparse structures, along with a similar isextended flag that the header had. There can be an indeterminate number of such extended_headers to describe a sparse file.
REGTYPE AREGTYPE These flags represent a regular file. In order to be compatible with older versions of tar, a typeflag value of AREGTYPE should be silently recognized as a regular file. New archives should be created using REGTYPE. Also, for backward compatibility, tar treats a regular file whose name ends with a slash as a directory.
LNKTYPE This flag represents a file linked to another file, of any type, previously archived. Such files are identified in Unix by each file having the same device and inode number. The linked-to name is specified in the linkname field with a trailing null.
SYMTYPE This represents a symbolic link to another file. The linked-to name is specified in the linkname field with a trailing null.
CHRTYPE BLKTYPE These represent character special files and block special files respectively. In this case the devmajor and devminor fields will contain the major and minor device numbers respectively. Operating systems may map the device specifications to their own local specification, or may ignore the entry.
DIRTYPE This flag specifies a directory or sub-directory. The directory name in the name field should end with a slash. On systems where disk allocation is performed on a directory basis, the size field will contain the maximum number of bytes (which may be rounded to the nearest disk block allocation unit) which the directory may hold. A size field of zero indicates no such limiting. Systems which do not support limiting in this manner should ignore the size field.
FIFOTYPE This specifies a FIFO special file. Note that the archiving of a FIFO file archives the existence of this file and not its contents.
CONTTYPE This specifies a contiguous file, which is the same as a normal file except that, in operating systems which support it, all its space is allocated contiguously on the disk. Operating systems which do not allow contiguous allocation should silently treat this type as a normal file.
A … Z These are reserved for custom implementations. Some of these are used in the GNU modified format, as described below.
Other values are reserved for specification in future revisions of the P1003 standard, and should not be used by any tar program.
The magic field indicates that this archive was output in the P1003 archive format. If this field contains TMAGIC, the uname and gname fields will contain the ASCII representation of the owner and group of the file respectively. If found, the user and group IDs are used rather than the values in the uid and gid fields.
For references, see ISO/IEC 9945-1:1990 or IEEE Std 1003.1-1990, pages 169-173 (section 10.1) for Archive/Interchange File Format; and IEEE Std 1003.2-1992, pages 380-388 (section 4.48) and pages 936-940 (section E.4.48) for pax - Portable archive interchange.