richgel999 / lzham_codec

Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++
Other
693 stars 71 forks source link

lzham_z_compress2 often returrns error #22

Open andr1972 opened 7 years ago

andr1972 commented 7 years ago

I write test how big enough must be buffer for good compression

void testSize(char *path, int size)
{   
    uLong IN_LEN;
    uLong in_len;
    uLong out_len;
    uLong new_len;
    IN_LEN = size;
    uint8 *in = (uint8 *)malloc(IN_LEN);
    out_len = compressBound(IN_LEN);
    uint8 *out = (uint8 *)malloc(out_len);
    FILE *fp = fopen(path, "rb");
    long sumin = 0;
    long sumout = 0;
    while (!feof(fp))
    {
        in_len = fread(in, 1, IN_LEN, fp);
        sumin += in_len;        
        int cmp_status = compress(out, &out_len, in, in_len);
        if (cmp_status != Z_OK)
        {
            printf("compress() failed dla size=%d %d %d\n", size, in_len, out_len);
        }
        sumout += out_len;
    }
    fclose(fp);
    free(in);
    free(out);
    printf("size=%d to %d compr=%f\n", size, sumout, (double)sumout / sumin);
}

void test()
{
    double dsize = 1000;
    while (dsize < 350000)
    {
        testSize("path_to_file_size_380kB", (int)dsize);
        dsize *= 1.1;
    }
}

Often cmp_status != Z_OK and is very slow for small buffers.

andr1972 commented 7 years ago

Was out_len too small:

        out_len = compressBound(IN_LEN);
        int cmp_status = compress(out, &out_len, in, in_len);

slow for small buffers because for small buffer it must init big dictionaries, I think.