Entity Test failed because of Unmatched Zone in ZonedDateTime between DB and ElasticSearch

Jie-Yang commented 8 years ago

After I install Entity-audit on Jhipster 3.4.0, All XXXResourceIntTest failed because lastModifiedDate and createdDate are not matched in following test. Also you can find the failure trace below. It seems that Entity has different time zones between DB and ElasticSearch.

FYI, if run ZonedDateTime.now(), I get London TimeZone which different from UTC from ElasticSearch.

` @Test @Transactional public void createAddress() throws Exception { int databaseSizeBeforeCreate = addressRepository.findAll().size();

    // Create the Address
    AddressDTO addressDTO = addressMapper.addressToAddressDTO(address);

    restAddressMockMvc
            .perform(post("/api/addresses").contentType(TestUtil.APPLICATION_JSON_UTF8)
                    .content(TestUtil.convertObjectToJsonBytes(addressDTO)))
            .andExpect(status().isCreated());

    // Validate the Address in the database
    List<Address> addresses = addressRepository.findAll();
    assertThat(addresses).hasSize(databaseSizeBeforeCreate + 1);
    Address testAddress = addresses.get(addresses.size() - 1);
    assertThat(testAddress.getName()).isEqualTo(DEFAULT_NAME);
    assertThat(testAddress.getAddress()).isEqualTo(DEFAULT_ADDRESS);

    // Validate the Address in ElasticSearch
    Address addressEs = addressSearchRepository.findOne(testAddress.getId());
    assertThat(addressEs).isEqualToComparingFieldByField(testAddress);
}`

The failure trace are:

java.lang.AssertionError: Expecting values: <[2016-06-23T09:30:33.072+01:00[Europe/London], 2016-06-23T09:30:33.072+01:00[Europe/London]]> in fields: <["createdDate", "lastModifiedDate"]> but were: <[2016-06-23T08:30:33.072Z[UTC], 2016-06-23T08:30:33.072Z[UTC]]> in <Address{id=5, name='AAAAA', address='AAAAA'}>. Comparison was performed on all fields at com.espion.ems.web.rest.AddressResourceIntTest.createAddress(AddressResourceIntTest.java:115)

deepu105 commented 8 years ago

I might not have time to work on this could you try to find out the root cause and provide a PR if possible?

mozartf commented 8 years ago

I got the same issue, it happened after I updated the code to support auditing entity. The two times are equivalent but the isEqualToComparingFieldByField method doesn't undestand timezones. Also, only happens in clases who inherits from ElasticsearchRepository, when we compare search data with database data.

Comparison was performed on all fields VenuesFeatureResourceIntTest.createVenuesFeature:109 Expecting values: <[2016-07-04T21:42:01.473-03:00[America/Cayenne], 2016-07-04T21:42:01.473-03:00[America/Cayenne]]> in fields: <["createdDate", "lastModifiedDate"]> but were: <[2016-07-05T00:42:01.473Z[GMT], 2016-07-05T00:42:01.473Z[GMT]]> in <VenuesFeature{id=6, name='AAAAA'}>. Comparison was performed on all fields VenuesFeatureResourceIntTest.updateVenuesFeature:194 Expecting values: <["anonymous", 2016-07-04T21:42:01.329-03:00[America/Cayenne]]> in fields: <["lastModifiedBy", "lastModifiedDate"]> but were: <[null, 2016-07-05T00:42:01.066Z[GMT]]> in <VenuesFeature{id=4, name='BBBBB'}>. Comparison was performed on all fields

deepu105 commented 8 years ago

I think this is happening only for the posgresDB for MySQL the tests are passing @Jie-Yang and @mozartf what DB are you guys using? can you post your yo-rc.json

Im not much of a Posgres user hence would appreciate any help here

deepu105 commented 8 years ago

ok, the failure might be related to presence of elasticsearch as well

deepu105 commented 8 years ago

@hipster-labs/developers does anyone have an idea about this?

mozartf commented 8 years ago

Hi, I faced this issue a while ago. But I can tell a few things I remember:

My database is MySql.
This is a timezone issue. The times in the test are off by three hours, exactly my timezone difference to GMT.
The time timezone information in the database seemed to be correct at the time; the problem was the time information stored in elasticsearch. The search is affected by this misinformation.
At the time I was playing with the generator-jhipster-entity-audit, that could have broken some tests.

I reverted the changes to a previous version of the project(previous Jhipster). So I'm not able to give you any more information, or reproduce the probles

Thanks for your help, any more questions that you have I'll try my best to answer because I'm interested in giving an upgrade on the Jhipster version of my project.

Regards, Mozart

Em 30 de ago de 2016 15:16, "Deepu K Sasidharan" notifications@github.com escreveu:

@hipster-labs/developers https://github.com/orgs/hipster-labs/teams/developers does anyone have an idea about this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hipster-labs/generator-jhipster-entity-audit/issues/37#issuecomment-243530623, or mute the thread https://github.com/notifications/unsubscribe-auth/AQbbm0Wa6ji3A3toi8nisVccIXAq3Ytoks5qlHNpgaJpZM4I8mIF .

Jie-Yang commented 8 years ago

Please find my .yo-rc.yml file below. In my case, h2 is used to run tests.

{ "generator-jhipster": { "jhipsterVersion": "3.4.0", "baseName": "ABC", "packageName": "com.es.abc", "packageFolder": "com/es/abc", "serverPort": "8080", "authenticationType": "session", "hibernateCache": "ehcache", "clusteredHttpSession": "no", "websocket": "no", "databaseType": "sql", "devDatabaseType": "h2Disk", "prodDatabaseType": "postgresql", "searchEngine": "elasticsearch", "buildTool": "gradle", "enableSocialSignIn": false, "rememberMeKey": "e677704271bdae415612315ae582fd5ec994282", "useSass": false, "applicationType": "monolith", "testFrameworks": [ "gatling" ], "jhiPrefix": "jhi", "enableTranslation": false }, "generator-jhipster-entity-audit": { "auditFramework": "custom" } }

santiagofalces commented 7 years ago

Hi, I faced this issue and I think the problem is that JpaRepository.save(entity) does not return the updated Audit fields (lastModifiedDate) after saving changes before doing the subsecuent myEntitySearchRepository.save(entity). So the changes on that fields do not reflect on Elasticsearch data. See: http://stackoverflow.com/questions/20382586/spring-data-repository-save-not-returning-instance-with-updated-audit-fields Also the timeZone problem when comparing datetimes is present, but that would be the smaller issue in this case.

I really don't have much more time to spend on this, but I spect this help somebody to address a solution.

(I didn't use this Module to generate de code, but the problem is the same)

pascalgrimaud commented 7 years ago

@santiagofalces : can you have a look at this https://github.com/jhipster/generator-jhipster/issues/4373 too ? Maybe it's related, but I'm not sure

santiagofalces commented 7 years ago

@pascalgrimaud For me, they are two separated issues, because in this case the DateTime values are different Instants. So this is not a DateTimeFormat error, it's storing an obsolete value for field "lastModifiedDate" when updating the search repository.

fdavaut commented 7 years ago

Hi,

I investigated a bit on this issue. Into the service addressServiceImpl, the addressSearchRepository.save(address) return an address object with good ZonedDateTime values for createdDate and lastModifiedDate.

public AddressDTO save(AddressDTO addressDTO) {
    log.debug("Request to save Adress : {}", addressDTO);
    Address address = addressMapper.addressDTOToAddress(addressDTO);
    address = addressRepository.save(address);
    AddressDTO result = addressMapper.addressToAddressDTO(address);
    addressSearchRepository.save(address);
    return result;
}

But if you add addressSearchRepository.findOne(address.getId()) just before the return, the object has bad values. The ZonedDateTime fields have 1 hour shift. It looks like a time zone error during a conversion.

jeyakumarsm commented 7 years ago

Hi, Looks like adding these lines in the generated ElasticSearchConfiguration.java working for me without any other changes in the JHipster generated code, please see if its the correct way to do. But I still see update entity tests failing.

objectMapper.configure(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE, false); objectMapper.configure(SerializationFeature.WRITE_DATES_WITH_ZONE_ID, true);

Full code:

`@Configuration public class ElasticSearchConfiguration {

@Bean
public ElasticsearchTemplate elasticsearchTemplate(Client client, Jackson2ObjectMapperBuilder jackson2ObjectMapperBuilder) {
    return new ElasticsearchTemplate(client, new CustomEntityMapper(jackson2ObjectMapperBuilder.createXmlMapper(false).build()));
}

public class CustomEntityMapper implements EntityMapper {

    private ObjectMapper objectMapper;

    public CustomEntityMapper(ObjectMapper objectMapper) {
        this.objectMapper = objectMapper;
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
        objectMapper.configure(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE, false);
        objectMapper.configure(SerializationFeature.WRITE_DATES_WITH_ZONE_ID, true);
    }

    @Override
    public String mapToString(Object object) throws IOException {
        return objectMapper.writeValueAsString(object);
    }

    @Override
    public <T> T mapToObject(String source, Class<T> clazz) throws IOException {
        return objectMapper.readValue(source, clazz);
    }
}

}`

deepu105 commented 7 years ago

could you submit this as a PR

Thanks & Regards, Deepu

On Tue, Dec 13, 2016 at 11:32 AM, jeyakumarsm notifications@github.com wrote:

Hi, Looks like adding these lines in the generated ElasticSearchConfiguration.java working for me without any other changes in the JHipster generated code, please see if its the correct way to do. But I still see update entity tests failing.

objectMapper.configure(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE, false); objectMapper.configure(SerializationFeature.WRITE_DATES_WITH_ZONE_ID, true);

Full code:

`@Configuration https://github.com/Configuration public class ElasticSearchConfiguration {

@Bean public ElasticsearchTemplate elasticsearchTemplate(Client client, Jackson2ObjectMapperBuilder jackson2ObjectMapperBuilder) { return new ElasticsearchTemplate(client, new CustomEntityMapper(jackson2ObjectMapperBuilder.createXmlMapper(false).build())); }

public class CustomEntityMapper implements EntityMapper {
private ObjectMapper objectMapper;

public CustomEntityMapper(ObjectMapper objectMapper) {
    this.objectMapper = objectMapper;
    objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
    objectMapper.configure(DeserializationFeature.ADJUST_DATES_TO_CONTEXT_TIME_ZONE, false);
    objectMapper.configure(SerializationFeature.WRITE_DATES_WITH_ZONE_ID, true);
}

@Override
public String mapToString(Object object) throws IOException {
    return objectMapper.writeValueAsString(object);
}

@Override
public <T> T mapToObject(String source, Class<T> clazz) throws IOException {
    return objectMapper.readValue(source, clazz);
}
}

}`

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hipster-labs/generator-jhipster-entity-audit/issues/37#issuecomment-266703406, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDlF9_DxP_dnlOcWymftaOzZDnYXa9Eks5rHnQpgaJpZM4I8mIF .

cbornet commented 7 years ago

I think this is just an issue with how you test the datetime value. 2016-07-04T21:42:01.473-03:00[America/Cayenne] and 2016-07-05T00:42:01.066Z[GMT] hold the same Instant but are not equal because ES doesn't preserve the timezone. So you can't compare with equals() you need to use ZonedDateTime::isEqual.

deepu105 commented 7 years ago

@cbornet could you please do a PR

cbornet commented 7 years ago

Well, that would be a fix in the main entity generator. What I don't understand is that ZonedDateTime+ES+not UTC Timezone should already be tested by Travis. cc @pascalgrimaud

jeyakumarsm commented 7 years ago

Hi @deepu105 first of all I m not sure what I have done is correct or not, but things are just working. And I m so new to JHipster and Generators, I might not be able to PR any time soon; Someone else can help this time, I can try from next time onwards. BTW, I have found out the root cause for update tests failures: tests started working when I used saveAndFlush in service implementation. So I think we have two solutions: 1) use saveAndFlush() instead of just save() in service/resource classes. 2) Invoke em.flush() just before findAll() call in update entity tests.

jeyakumarsm commented 7 years ago

Sorry, I take back the second solution, only saveAndFlush works on update entity tests.

day0ops commented 7 years ago

Any update on this ?

DanielFran commented 2 years ago

Closing since Friday xed with latest version

jhipster / generator-jhipster-entity-audit

Entity Test failed because of Unmatched Zone in ZonedDateTime between DB and ElasticSearch #37