search

scylladb / scylla-jmx

Scylla JMX proxy
GNU Affero General Public License v3.0
29 stars 52 forks source link

scylla-jmx does not look for new CF after restart #63

Closed amnonh closed 5 years ago

amnonh commented 5 years ago

When looking for a column family, the jmx should check with the API if it exists, and if it does it would return it. (originally the jmx would use pulling, but that was too load intensive on the system).

It seems that after the latest changes in scylla-jmx it's no longer the case, and now scylla-jms load the tables that are there on startup and does not update it later.

This means that most of the tables are not available.

amnonh commented 5 years ago

@elcallio I didn't know which pioter to cc

elcallio commented 5 years ago

Just tested this. It refreshes fine for me. The mbean client must do a query for this to happen, but that has always been the case.

bentsi commented 5 years ago

is this going to be fixed for 3.1? without this, nodetool toppartition will not work @slivne @roydahan @aleksbykov

slivne commented 5 years ago

@amnonh - I don't understand - did the previous version behave differently ?

If so how - please note bentsi is complaining that toppartition will not work for some reason .

elcallio commented 5 years ago

Behaviour has not changed in like 2 years. And even then it was the same. However, nodetool topppartions might have. It makes a query that does not match the column family scanner test. It checks type=*ColumnFamilies. This might be a 3.1 update issue that was not discovered before.

amnonh commented 5 years ago

@slivne I open this issue after talking to Bentsi, for some reason, nodetool toppartition does not find tables (cf) that were created after the jmx was started.

To verify this, restarting the jmx makes it work.

toppartition is a new command, maybe it uses a different path, maybe it's the changes in the jmx registrations that were made with the move to 3.0

amnonh commented 5 years ago

Toppartition use the nodeprobe.getCfsProxy which suppose to be the standrad way of fetching columnfamily.

The name it uses there is: org.apache.cassandra.db:type=*" + type +",keyspace=" + ks + ",columnfamily=" + cf

What makes me thing the issue with the mbean server is that the initilization order count.

So if the jmx starts when the cf already exists, it's ok, if the cf is created after the jmx was started, it does not work.

elcallio commented 5 years ago

The problem, again, is the query org.apache.cassandra.db:type=*" + type. This does not match the refresh trigger. I'll send a patch, but there might be some other lingering issues with stuff that as been added onto here. I get some exception that should not be.

bentsi commented 5 years ago

@elcallio please update us when the patch will be in

elcallio commented 5 years ago

I sent the patch yesterday. When it will be in, or if anyone even notices it I cannot say.

bentsi commented 5 years ago

@elcallio who should check the patch?

elcallio commented 5 years ago

Maybe @amnonh. Or @haaawk

bentsi commented 5 years ago

@elcallio what is the status with this patch?

bentsi commented 5 years ago

any progress here?

slivne commented 5 years ago

"[PATCH scylla-jmx 2/2] APIMBeanServer: Handle nodeprobe wildcard queries in CF refresh"