Random malloc: *** error for object 0x7fca304091c0: incorrect checksum for freed object - object was probably modified after being freed.

[lzuwei]

I have been getting some weird resource release errors on the Aerospike C Client on MacOSX Yosemite running a Vagrant build of Aerospike 3.5.14.

I am running the following codes to insert into an lmap in Aerospike 3.5.14 Community Edition with C Client 3.1.18

//add them into database
    std::map<newton::InternalLabelType, int>::const_iterator end = lbl_count.end();
    for(std::map<newton::InternalLabelType, int>::const_iterator it = lbl_count.begin(); it != end; ++it) {
        as_ldt lmap;
        as_ldt_init(&lmap, "images", AS_LDT_LMAP, NULL);

        as_string rec_guid;
        as_string_init(&rec_guid, (char*)guid.c_str(), false);

        //create key based on cluster number
        as_key key;
        as_key_init_int64(&key, "iq", "clusters", it->first);

        as_integer count;
        as_integer_init(&count, it->second);

        as_policy_apply policy;
        as_policy_apply_init(&policy);
        policy.key = AS_POLICY_KEY_SEND;

        as_error err;
        if(aerospike_lmap_put(m_as, &err, &policy, &key, &lmap,
                              (const as_val*)&rec_guid, (const as_val*)&count) != AEROSPIKE_OK) {
            fprintf(stderr, "aerospike_lmap_put() returned %d - %s", err.code, err.message);
        }

        as_key_destroy(&key);
        as_ldt_destroy(&lmap);
        as_string_destroy(&rec_guid);
        as_integer_destroy(&count);
    }

An other function calls the loop which usually run around 500 to 1000 times lbl_count is a map that looks like this

{
1: 3,
2: 55,
3: 100
}

The insertion runs normal and breaks randomly at different points issuing an error: malloc: * error for object 0xc6a3970: pointer being freed was not allocated

Debugging through lldb shows that the error occurred in aerospike_lmap_put(), Here is the context of the code

// =======================================================================
// PUT
// =======================================================================
as_status aerospike_lmap_put(
    aerospike * as, as_error * err, const as_policy_apply * policy,
    const as_key * key, const as_ldt * ldt,
    const as_val * mkey, const as_val * mval)
{

//... codes 
    as_val* p_return_val = NULL;
    aerospike_key_apply(
        as, err, policy, key, DEFAULT_LMAP_PACKAGE, LDT_MAP_OP_PUT,
        (as_list *)&arglist, &p_return_val);

    as_arraylist_destroy(&arglist);

    if (ldt_parse_error(err) != AEROSPIKE_OK) {
        return err->code;
    }

    if (!p_return_val) {
        return as_error_set(err, AEROSPIKE_ERR_LDT_INTERNAL,
                "no value returned from server");
    }
    int64_t ival = as_integer_getorelse(as_integer_fromval(p_return_val), -1);
    as_val_destroy(p_return_val); //error happens here 

    if (ival == -1) {
        return as_error_set(err, AEROSPIKE_ERR_LDT_INTERNAL,
                "value returned from server not parse-able");
    }

    return err->code;
}

The error happens at line 97 as_val_destroy(p_return_val); where the p_return_val is attempted to be freed.

In the as_val_destroy() code, the error is traced to line 162

as_val * as_val_val_destroy(as_val * v)
{
    if ( v == NULL || !v->count ) return v;
    // if we reach the last reference, call the destructor, and free
    if ( 0 == cf_atomic32_decr(&(v->count)) ) {
        as_val_destroy_callbacks[ v->type ](v);     
        if ( v->free ) {
            cf_free(v); //error happens here
        }
        v = NULL;
    }
    return v;
}

I have tested the code on ubuntu 14.04 gcc 4.8.2, and it completes without any issues. If anyone can provide some help to this will be greatly appreciated!

[wchu-citrusleaf]

We are investigating. Thanks