Open metdos opened 9 years ago
Hey @jasonmp85,
I examined the bug thoroughly. It seems the problem (or a very close problem) may lead to some other bugs too.
First, let me illuminate the problem. Basically, our approach does the following:
INSERT
to the temporary table, via the trigger, INSERT
to the distributed table.COPY
to the temporary table. The COPY
command triggers the INSERT
to the distributed table.Now, how this bug happens as follows:
COPY
command, the stopped worker nodes is marked as STATE_INACTIVE
. Thus, it does not try to INSERT
to the shard placements that are on the stopped worker node.STATE_FINALIZED
. (I guess, it is due to MVCC )COPY
fails, the transaction that COPY
is executed inside is rolled backed. STATE_INACTIVE
marked as STATE_FINALIZED
again. This problem can also be observed in different forms. Such as the following:
COPY
command, the stopped worker nodes is marked as STATE_INACTIVE
.STATE_FINALIZED
. (I guess, it is due to MVCC ). Thus, concurrent queries can read from STATE_INACTIVE
shard placements.I tried to find some ways to handle this bug. What do you think about them? Which one should I follow? Or can you suggest any other way?
COPY
makes things difficult to handle. Mainly it is because we are on a single transaction on the master node, but, we execute too many transactions on the worker nodes. Rolling back all the INSERT
s is not simple. So, one approach could be to change COPY
to consecutive INSERT
s, which in turn may decrease the overall performance. This may require a lot of code change too. (Handling COPY parameter etc.)copy_to_distributed_table
, use some kind of DO BLOCK or pl/pgsql functions instead of this part. Then, write a code to iterate over the file. On each iteration, copy a single line from the file and save it to a temporary file. Lastly, COPY
from that temporary file. This method incurs high overhead, but, at least we do not lose the gain of using COPY
or we don't need to change too many lines of codes. We need to consider newline character to parse the file, which can be done easily.COPY
command, we can bypass this problem.Hey @jasonmp85,
Changing the value of ON_ERROR_ROLLBACK
has no effect on our case. I read this post which explains the effects of the parameter in detail. Despite what is written on the post, I tested with changing the value of it, but as I expected the results didn't change.
To summarize what I observed:
ON_ERROR_ROLLBACK
is designed for error handling during execution consecutive commands inside a transaction. However, we only have a single transaction associated with \COPY
command. copy_to_insert
that it starts a transaction. By that way, the metadata is updated inside that transaction and quitting the script does not lead to this bug. However, as explained here, it is not possible to start a transaction inside a PL/pgSQL function.So, it seems that we cannot solve this problem with playin ON_ERROR_ROLLBACK
.
If I shutdown(ctrl + c) copy_to_distributed_table while it is working, it fails to mark shards on closed node as inactive. If I wait copy_to_distributed_table to complete, then it marks shards inactive on closed node as properly.
Here are steps to replicate problem;
/usr/local/pgsql/bin/copy_to_distributed_table -CH -n NULL customer_reviews_1998.csv customer_reviews
select count(*) from customer_reviews;
select count(*) from customer_reviews;
and see results are different.