Adding a new column of type list<String> and renaming existing column at the same time (in a single entity) made the schema go into a weird state

[ChandanNM]

The sync mechanism failed to create the columns specified in the entity because of which the REST APIs for GET and POST failed. Then reverting the changes i.e removing added columns failed to bring up application with the error :

Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Column was not found in table

Only option was to reset the keyspace

[valchkou]

Please provide details:

• old entity
• new entity

[rahulsrivastava71]

drop the the existing keyspace and then try again to sync with your changes, it will be done. And yes please send the old and new entity as asked by eugene.

On Tue, Aug 25, 2015 at 9:42 PM, ChandanNM notifications@github.com wrote:

The sync mechanism failed to create the columns specified in the entity because of which the REST APIs for GET and POST failed. Then reverting the changes i.e removing added columns failed to bring up application with the error :

Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Column was not found in table

Only option was to reset the keyspace

— Reply to this email directly or view it on GitHub https://github.com/valchkou/cassandra-driver-mapping/issues/67.

[ChandanNM]

Yes, that's what I had to do. But this was on our staging cloud env and not local development. So I cannot drop the keyspace often if the sync wouldn't work.

Realized after my recent set of changes that the sync itself is not working on our staging cloud env - Creates keyspace and tables the first time if doesn't exist but the subsequent syncs fail on any new columns/indexes add. However its weird that the same deployment works on local docker dev and also production cloud env. Cassandra version is 2.1.7 Wondering if force syncing every-time might help ? - even though no entity modifications

FYI - My old entity :

package com.cisco.collab.terminus.db.entity.beans;

import java.util.List; import java.util.Map; import java.util.UUID;

import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.Table;

import org.apache.commons.lang3.builder.EqualsBuilder; import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.commons.lang3.builder.ToStringBuilder;

@Entity @Table(name="customer") public class Customer {

@Id
@Column(name = "pkid")
private UUID pkid;

@Column(name = "name")
private String name;

@Column(name = "billingaddress")
private Map<String, String> billingAddress;

@Column(name = "orders")
private List<UUID> orders;

@Column(name = "sites")
private List<UUID> sites;

@Column(name = "reseller")
private String reseller;

@Column(name = "pstncarrier")
private String pstnCarrier;

@Column(name = "carriercustomerid")
private String carrierCustomerId;

public UUID getPkid() {
    return pkid;
}

public void setPkid(final UUID pkid) {
    this.pkid = pkid;
}

public String getName() {
    return name;
}

public void setName(final String name) {
    this.name = name;
}

public Map<String, String> getBillingAddress() {
    return billingAddress;
}

public void setBillingAddress(final Map<String, String> billingAddress) {
    this.billingAddress = billingAddress;
}

public List<UUID> getOrders() {
    return orders;
}

public void setOrders(final List<UUID> orders) {
    this.orders = orders;
}

public List<UUID> getSites() {
    return sites;
}

public void setSites(final List<UUID> sites) {
}

public String getReseller() {
    return reseller;
}

public void setReseller(final String reseller) {
    this.reseller = reseller;
}

public String getPstnCarrier() {
    return pstnCarrier;
}

public void setPstnCarrier(final String pstnCarrier) {
    this.pstnCarrier = pstnCarrier;
}

public String getCarrierCustomerId() {
    return carrierCustomerId;
}

public void setCarrierCustomerId(final String carrierCustomerId) {
    this.carrierCustomerId = carrierCustomerId;
}

@Override
public String toString() {
    return ToStringBuilder.reflectionToString(this);
}

@Override
public int hashCode() {
    return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public boolean equals(final Object other) {
    return EqualsBuilder.reflectionEquals(this, other);
}

}

My new entity :

package com.cisco.collab.terminus.db.entity.beans;

import java.util.List; import java.util.Map; import java.util.UUID;

import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.Table;

import org.apache.commons.lang3.builder.EqualsBuilder; import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.commons.lang3.builder.ToStringBuilder;

@Entity @Table(name="customer") public class Customer {

@Id
@Column(name = "pkid")
private UUID pkid;

@Column(name = "name")
private String name;

@Column(name = "billingaddress")
private Map<String, String> billingAddress;

@Column(name = "orders")
private List<UUID> orders;

@Column(name = "sites")
private List<UUID> sites;

@Column(name = "dids")
private List<String> dids;

@Column(name = "reseller")
private String reseller;

@Column(name = "pstncarrier")
private String pstnCarrier;

@Column(name = "carriercustomerid")
private String carrierCustomerId;

public UUID getPkid() {
    return pkid;
}

public void setPkid(final UUID pkid) {
    this.pkid = pkid;
}

public String getName() {
    return name;
}

public void setName(final String name) {
    this.name = name;
}

public Map<String, String> getBillingAddress() {
    return billingAddress;
}

public void setBillingAddress(final Map<String, String> billingAddress) {
    this.billingAddress = billingAddress;
}

public List<UUID> getOrders() {
    return orders;
}

public void setOrders(final List<UUID> orders) {
    this.orders = orders;
}

public List<UUID> getSites() {
    return sites;
}

public void setSites(final List<UUID> sites) {
}

public List<String> getDids() {
    return dids;
}

public void setDids(final List<String> dids) {
    this.dids = dids;
}

public String getReseller() {
    return reseller;
}

public void setReseller(final String reseller) {
    this.reseller = reseller;
}

public String getPstnCarrier() {
    return pstnCarrier;
}

public void setPstnCarrier(final String pstnCarrier) {
    this.pstnCarrier = pstnCarrier;
}

public String getCarrierCustomerId() {
    return carrierCustomerId;
}

public void setCarrierCustomerId(final String carrierCustomerId) {
    this.carrierCustomerId = carrierCustomerId;
}

@Override
public String toString() {
    return ToStringBuilder.reflectionToString(this);
}

@Override
public int hashCode() {
    return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public boolean equals(final Object other) {
    return EqualsBuilder.reflectionEquals(this, other);
}

}

[valchkou]

Thanks for the code sample. I was not able to reproduce the issue. Changes were synched fine.
Which columns were missing or were not renamed in your case ? let me guess that you had 2 clients running at the same time with old and new Customer entity. The first made new schema the second one reset the schema back to old. So finally you got one of clients blown.

[rahulsrivastava71]

Dear Eugene , Can you please tell which version of Datastax you are using, in your mapper.

On Wed, Sep 2, 2015 at 4:23 AM, Eugene notifications@github.com wrote:

Thanks for the code sample. I was not able to reproduce the issue. Changes were synched fine.

Which columns were missing or were not renamed in your case ? let me guess that you had 2 clients running at the same time with old and new Customer entity. The first made new schema the second one reset the schema back to old. So finally you got one of clients blown.

— Reply to this email directly or view it on GitHub https://github.com/valchkou/cassandra-driver-mapping/issues/67#issuecomment-136885390 .

[valchkou]

Look at your mapper ver. The same version of driver

[ChandanNM]

Since its a cloud environment we do have 2 instances running. Whenever we deploy the new one on our staging system - 2 instances of the newly deployed app get created, all the traffic is then switched to the new ones and the old ones are removed. I think this would be the normal behavior in any cloud environment to account for high availability. Would the sync mechanism fail in such scenarios ? As I said before this does work in local docker env where we just have one instance of the app deployed.

[valchkou]

Cassandra is schema based DB. So you have to do stop-the-world pause while applying schema changes. you can't have old and new running at the same time. You have to completely stop old nodes before switching to new if schema changes are involved.

Another option would be to disable sync at all and manually apply schema changes. But still you have to stop old nodes , apply schema script and start new.

[ChandanNM]

Ok thanks Eugene. We are trying out couple of options. I will update here based on the results

[ChandanNM]

Eugene, after changing the app instance to one instead of two, the schema behaved well for a while till I hit another issue today after adding an attribute of type : Set

Below is the error :

OUT [CONTAINER] ing.AutoReconfigurationServletContainerInitializer INFO Initializing ServletContext with Auto-reconfiguration ApplicationContextInitializers 2015-09-17T22:29:29.90+0000 [App/0] OUT [CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] INFO Spring WebApplicationInitializers detected on classpath: [org.glassfish.jersey.server.spring.SpringWebApplicationInitializer@2da43e96] 2015-09-17T22:29:32.76+0000 [App/0] OUT 22:29:32.755 [localhost-startStop-1] INFO (CassandraUtils.java:49) - Keyspace - pstn_marketplace created successfully. 2015-09-17T22:29:32.92+0000 [App/0] OUT [CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] SEVERE Exception sending context initialized event to listener instance of class com.cisco.collab.terminus.listener.SchemaSyncListener 2015-09-17T22:29:32.92+0000 [App/0] OUT com.datastax.driver.core.exceptions.SyntaxError: line 0:-1 mismatched input '' expecting '<' 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.datastax.driver.core.exceptions.SyntaxError.copy(SyntaxError.java:35) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:269) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:183) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.datastax.driver.mapping.schemasync.SchemaSync.sync(SchemaSync.java:51) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.cisco.collab.terminus.utils.CassandraUtils.schemaSync(CassandraUtils.java:73) 2015-09-17T22:29:32.92+0000 [App/0] OUT at com.cisco.collab.terminus.listener.SchemaSyncListener.contextInitialized(SchemaSyncListener.java:29) 2015-09-17T22:29:32.92+0000 [App/0] OUT at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729)

Entity :

package com.cisco.collab.terminus.db.entity.beans;

import java.util.List; import java.util.Map; import java.util.Set; import java.util.UUID;

import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.Id; import javax.persistence.Index; import javax.persistence.Table;

import org.apache.commons.lang3.builder.EqualsBuilder; import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.commons.lang3.builder.ToStringBuilder;

@Entity @Table(name="customer", indexes = { @Index(name="entity_name_idx", columnList="name" ) }) public class Customer {

@Id
@Column(name = "pkid")
private UUID pkid;

@Column(name = "name")
private String name;

@Column(name = "billingaddress")
private Map<String, String> billingAddress;

@Column(name = "orders")
private List<UUID> orders;

@Column(name = "sites")
private List<UUID> sites;

@Column(name = "reseller")
private String reseller;

@Column(name = "pstncarrier")
private String pstnCarrier;

@Column(name = "customerid")
private String customerId;

@Column(name = "dids")
private Set<String> dids;

public UUID getPkid() {
    return pkid;
}

public void setPkid(final UUID pkid) {
    this.pkid = pkid;
}

public String getName() {
    return name;
}

public void setName(final String name) {
    this.name = name;
}

public Map<String, String> getBillingAddress() {
    return billingAddress;
}

public void setBillingAddress(final Map<String, String> billingAddress) {
    this.billingAddress = billingAddress;
}

public List<UUID> getOrders() {
    return orders;
}

public void setOrders(final List<UUID> orders) {
    this.orders = orders;
}

public List<UUID> getSites() {
    return sites;
}

public void setSites(final List<UUID> sites) {
    this.sites = sites;
}

public String getReseller() {
    return reseller;
}

public void setReseller(final String reseller) {
    this.reseller = reseller;
}

public String getPstnCarrier() {
    return pstnCarrier;
}

public void setPstnCarrier(final String pstnCarrier) {
    this.pstnCarrier = pstnCarrier;
}

public String getCustomerId() {
    return customerId;
}

public void setCustomerId(final String customerId) {
    this.customerId = customerId;
}

public Set<String> getDids() {
    return dids;
}

public void setDids(Set<String> dids) {
    this.dids = dids;
}

@Override
public String toString() {
    return ToStringBuilder.reflectionToString(this);
}

@Override
public int hashCode() {
    return HashCodeBuilder.reflectionHashCode(this);
}

@Override
public boolean equals(final Object other) {
    return EqualsBuilder.reflectionEquals(this, other);
}

}

[ChandanNM]

FYI - The new property I added is - private Set dids.

Dependencies defined in my build.gradle -

compile "com.datastax.cassandra:cassandra-driver-core:2.1.7" compile "com.valchkou.datastax:cassandra-driver-mapping:2.1.5"

[valchkou]

The latest dependency is compile "com.valchkou.datastax:cassandra-driver-mapping:2.2.0-rc1"

you shouldn't do this at all: compile "com.datastax.cassandra:cassandra-driver-core:2.1.7" cassandra-driver-mapping automatically gets matching datastax driver dependency.

Anyway it's not going to help as you continue hit same DB from different places with different entity versions.