tomholub
commented
7 years ago Downgraded all nodes to v1.0.6 and the fourth node is happy now, accepting and serving requests.
Closed tomholub closed 6 years ago
tomholub
commented
7 years ago Downgraded all nodes to v1.0.6 and the fourth node is happy now, accepting and serving requests.
a-robinson
commented
7 years ago Thanks for the details, @tomholub! Getting the full dump of logs is fantastic. Just to make sure I'm understanding you, all of the nodes, including the fourth one, were running 1.1-beta.20170928?
Do you have the logs from when the fourth node died or from the other nodes? And can you take screenshots of the Replication graphs from the admin UI (at path /#/cluster/all/replication)?
tomholub
commented
7 years ago Yes, during the time that this was happening, all nodes were running 1.1-beta.20170928, including the fourth one. I retrieved the logs from n-4 through another node's dashboard during the time that n-4 was running, in between crashes. I have copied the full log until the end of fourth node - or so I believe. You will see two WARNING | RUNNING IN INSECURE MODE! in the logs, the second one is there because node was restarted after a crash.
Unfortunately, that cluster was a testing one and I didn't retain any records of it.
Note that this was tested on 1GB ram machines (I understand that 2GB is recommended). There were other services running on these nodes, and although other services were idle, I wonder if the 4th node was just running out of memory / crashing for that reason.
Either way, that should not affect the remaining nodes in the cluster. Interestingly, the stable 1.0.6 didn't show these issues.
tomholub
commented
7 years ago Also, looking at my records, there was a replication factor of 3 set this way:
echo "num_replicas: 3" > zone.yaml
cockroach zone set .default --insecure -f zone.yaml(part of a node starting script)
a-robinson
commented
7 years ago Alright, I've got a repro and will make sure we get a fix in soon. It looks like we're hitting a case along the lines of https://github.com/cockroachdb/cockroach/issues/17971. Thanks a ton for the report!
dianasaur323
commented
7 years ago awesome, I'm glad we got this fixed! By the way, @tomholub it looks like you are running a globally distributed cluster, which is something we caution people about since there are still several really exciting improvements we are making to the product that would make that experience better. @a-robinson is very much leading the charge on those features. How has that been working for you so far? Would love to get your general thoughts to see what we could improve for you?
tomholub
commented
7 years ago Flexible write consistency per tx, less memory, more stability.
Thanks for this db, it has potential!
I thought roach may be a perfect fit for us, as we don't have lots of throughput and don't require a whole lot of optimization, just a low maintenance geo-distributed db to reduce latency for our users across the globe.
Roach does that for reads, but the writes are painfully slow in a geo-distributed db.
Here are some of my notes, kind of my x-mas wishlist for a perfect db:
They say that it's worth to solve a problem if potential users are already solving it in their own painful way. We ended up using MariaDB Galera cluster for now - painful indeed, but workable.
We can run these on $5 nodes instead of the $20 roach nodes. That means we can afford more nodes, and get closer to our users, globally.
Geo-distributed writes are slower, though comparable to roach. We resorted to developing primitive eventually-consistent writes in our app code like this:
Consistent writes:
app -> dbEventually consistent writes:
app -> redisapp -> write queuewrite queue -> db (Last write wins. Dropped when db rejects it - eg unique constraint)All reads:
app <- redis (includes stuff that's in db write queue)app <- dbIt allows us to do fast local reads as well as writes, while still having option to do consistent strong writes when needed.
But really, we're replicating what I think the db should be able to do itself. If roach could do something like that, with a bit less memory, it would be a solid choice.
dianasaur323
commented
7 years ago amazing, thanks for the detailed feedback @tomholub
Regarding making globally deployed clusters more performant, we have several ideas floating around that might interest you. 1) allowing non-voting replicas to service potentially stale reads, 2) setting a safe max timestamp in which a client can make a safe read of a non-voting replica, 3) geo-partitioning coming up in 1.2 in which you can pin rows / ranges to specific datacenters.
Supporting eventual consistency on writes is almost an existential question for us. We've erred towards trying to find ways to make writes consistent, but faster. If we allow eventual consistency, you would end up having many of the issues that other databases have. It's a catch-22! Have you considered trying to use zone configs at a greater granularity? For example, if you don't care as much about resiliency, you could theoretically have a Singapore table where all three replicas live in Singapore, and you would be able to serve faster reads and writes for Singapore data to Singapore users. Not really the best production setup, but a potentially interesting thought experiment. This should get better when we build out row-level partitioning, which will allow this configuration at the row level.
The cost efficiency metric you provided is super helpful - really useful to see that stark contrast. We should work on making that better...
tomholub
commented
7 years ago We also need Germans to read data about Singaporean users (from German server, but not necessarily in real time).
A weak write in Singapore needs to be immediately readable in Singapore. It's ok if it takes 10 seconds to be readable in Germany (for us).
In other words - we expect to see a lot of global dirty reads in between datacenters (writes may take a bit of delay to sync), but want to be able to configure immediate sync / no dirty reads within a locality. We'd also need to retain option to use the current write mechanism when we need a strong global write. Pack that onto a 1gb machine reliably and we are golden.
Approach 1 - integrated queue as an option
Leave everything as is, and add a write-queue mechanism. I can either do a standard roach write (strong consistency, large latency) or I can do a delayed (queued) roach write. Delayed write will reflect immediately on the same node (from memory cache), but the actual write will be performed at a later time, as a standard roach write, just processed from a queue. You could only allow these when editing columns with no constraints, so that the txs cannot fail (not sure if feasible - postgres txs are complicated). Or maybe special key sequence like IGNORE ON CONFLICT in case I still want to queue-update columns that have constraints.
Approach 2 - locality has authority over rows it created - configurable option This would be a very radical approach. Not sure if possible within postgres mentality (sounds like nosql to me). If a certain row was created in Singapore locality, then Singapore nodes can write to it "selfishly" without regard to what nodes in other localities think. Other nodes can write to Singaporean-owned rows through standard roach write process. If the two conflict, Singapore always wins for that row. We don't really do multi-row updates, I'm not sure how to handle these with a system like that. I'd also like to be able to transfer ownership of rows if I find out that the user has moved. Damn, that would be a cool database.
a-robinson
commented
6 years ago Fixed as part of https://github.com/cockroachdb/cockroach/issues/17971
dianasaur323
commented
6 years ago @tomholub I just read through this issue again since it came through my email with the most recent fix. Apologies for the lack of follow up. Thank you for the feedback and thoughts - you should definitely consider playing around with our next new feature (geo-partitioning) coming up in our new release. It won't let you do those eventual consistency writes, but it will allow you to have much better control over where your rows live.
tomholub
commented
6 years ago Thanks for the update
1) a cluster of 3 nodes spanning New York, Germany and Singapore.
2) I added a dozen records and load-tested the NY node with reads. 3) While NY under workload, I added a fourth node next to it. Now we have following localities:
My hope and expectation was that the data would replicate to the fourth node, and maybe it would even start transparently helping the other node in the same datacenter (Elasticsearch style)
Instead, the moment the node started, throughput went down the drain and stayed there, until the node that joined died a few minutes later:
All nodes running 1.1-beta.20170928
Eventually, after a few similar cycles, I was able to get fourth node be a part of the cluster without affecting the neighbor, but it will not accept sql connections, alternate between healthy and suspect, and die within 10 minutes of starting.
I'm working with 3 tables, < 10 rows and maybe 10kb data. Also tried to let the cluster rest - but fourth node will not become functional.
These are the logs from node 4 colocated with the NY node (different machine on same datacenter):
Sorry for the big dump, hard to say which logs are important.