search

thulab / iotdb

This repository is ReadOnly now. please go to https://github.com/apache/incubator-iotdb
Apache License 2.0
19 stars 0 forks source link

maybe a concurrent error when multi-client insert a new data into memtable #520

Closed jixuan1989 closed 5 years ago

jixuan1989 commented 5 years ago

I noticed that we do not obtain any lock when write data in memtable.

Suppose two client send data to IoTDB, one writes d1.s1 and meanwhile the other writes d1.s2. If both of them run memTableMap.put(deltaObject, new HashMap<>());, then we will lost some data.

see AbstractMemTable.java:

    private IMemSeries createIfNotExistAndGet(String deltaObject, String measurement, TSDataType dataType) {
        if(!memTableMap.containsKey(deltaObject)) {
            memTableMap.put(deltaObject, new HashMap<>());
        }
        Map<String, IMemSeries> memSeries = memTableMap.get(deltaObject);
        if(!memSeries.containsKey(measurement)) {
            memSeries.put(measurement, genMemSeries(dataType));
        }
        return memSeries.get(measurement);
    }
jixuan1989 commented 5 years ago

@liuruiyiyang can you try to reproduce the scenario? @liukun4515 can you confirm it logically? By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

liukun4515 commented 5 years ago

A thread can call the interface of memtable, it should acquire the lock in the corresponding filenode first. @jixuan1989

liukun4515 commented 5 years ago

By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

The data is written into the memtable, which will be flushed in a background thread. At the same time, we will give a new memtable which is used to store newly written data.

xingtanzjr commented 5 years ago

By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

The data is written into the memtable, which will be flushed in a background thread. At the same time, we will give a new memtable which is used to store newly written data.

By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

The data is written into the memtable, which will be flushed in a background thread. At the same time, we will give a new memtable which is used to store newly written data.

So this problem doesn't exist, right ?

jixuan1989 commented 5 years ago

But I think it is a coarse-grained implementation. It limits the concurrency. Anyway, current implementation is correct.

liukun4515 commented 5 years ago

By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

The data is written into the memtable, which will be flushed in a background thread. At the same time, we will give a new memtable which is used to store newly written data.

By the way, when a user write data, and at the same time, a flushing operation occurs, what will happen?

The data is written into the memtable, which will be flushed in a background thread. At the same time, we will give a new memtable which is used to store newly written data.

So this problem doesn't exist, right ?

yes @xingtanzjr

liukun4515 commented 5 years ago

But I think it is a coarse-grained implementation. It limits the concurrency. Anyway, current implementation is correct.

The method of concurrent control must limit the concurrency, but we should do something else synchronously for each filenode processor before data to be written into the memtable. @jixuan1989