[regression]: migrations fail

nbittich commented 2 years ago

Step to reproduce:

[ ] checkout CH project (https://github.com/lblod/app-contact-hub)
[ ] Switch docker image for virtuoso
[ ] fetch private migrations files in CH ACC (config/migrations/private-data/**)
[ ] delete data/db
[ ] run docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d

Expected result:

All migrations running and status DONE

Actual result:

20210901153815-worship-local.ttlfails with error:

igrations_1                                                | E, [2021-12-07T11:21:48.067865 #12] ERROR -- : Invalid Turtle file 20210901153815-worship-local.ttl
migrations_1                                                | /usr/local/lib/ruby/2.5.0/net/protocol.rb:181:in `rbuf_fill': too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 47381480942380, last used 1638876108.0677445 seconds ago (Net::HTTP::Persistent::Error)
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/protocol.rb:157:in `readuntil'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/protocol.rb:167:in `readline'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http/response.rb:40:in `read_status_line'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http/response.rb:29:in `read_new'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1494:in `block in transport_request'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1464:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:964:in `block in request'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:662:in `connection_for'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:958:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:696:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:344:in `response'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:305:in `query'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:22:in `query'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:26:in `update'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:8:in `block in included'
migrations_1                                                |   from /usr/src/app/ext/web.rb:129:in `batch_insert'
migrations_1                                                |   from /usr/src/app/ext/web.rb:78:in `execute!'
migrations_1                                                |   from /usr/src/app/ext/web.rb:151:in `block in execute_migrations'
migrations_1                                                |   from /usr/src/app/ext/web.rb:149:in `each'
migrations_1                                                |   from /usr/src/app/ext/web.rb:149:in `execute_migrations'
migrations_1                                                |   from /usr/src/app/ext/web.rb:214:in `boot'
migrations_1                                                |   from /usr/src/app/ext/web.rb:217:in `<top (required)>'
migrations_1                                                |   from web.rb:87:in `require_relative'
migrations_1                                                |   from web.rb:87:in `<main>'

Database logs:

triplestore_1                                               | 11:21:50 ERRS_0 HY008 SR189 Async statement killed by SQLCancel.
triplestore_1                                               | 11:21:50 ERRS_0 S1T00 {CLI.. Client cancelled or disconnected
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 S1T00 {CLI.. Client cancelled or disconnected
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 S1T00 {CLI.. Client cancelled or disconnected
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 S1T00 {CLI.. Client cancelled or disconnected
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 S1T00 {CLI.. Client cancelled or disconnected
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:21:50 ERRS_0 40001 SR337 Transaction aborted due to async rollback in cluster
triplestore_1                                               | 11:26:53 ERRS_0 01V01 QW004 Incompatible types VARCHAR (182) and INTEGER (189) in COALESCE for coalesce_ret and <constant>
triplestore_1                                               | 11:26:53 ERRS_0 01V01 QW004 Incompatible types VARCHAR (182) and INTEGER (189) in := for coalesce_ret and <constant>
triplestore_1                                               | 11:26:53 ERRS_0 01V01 QW004 Incompatible types VARCHAR (182) and INTEGER (189) in COALESCE for coalesce_ret and <constant>
triplestore_1                                               | 11:26:53 ERRS_0 01V01 QW004 Incompatible types VARCHAR (182) and INTEGER (189) in := for coalesce_ret and <constant>

nvdk commented 2 years ago

@nbittich I believe this issue should be resolved on redpencil/virtuoso:1.0.0

nbittich commented 2 years ago

@nvdk I think I've got that bug when we tried to upgrade to 1.0.0; so i'm not sure it is indeed fix, I can try again and let you know

nbittich commented 2 years ago

@nvdk I just tried again & same problem:

`rbuf_fill': too many connection resets (due to Net::ReadTimeout - Net::ReadTimeout) after 0 requests on 47221829094260, last used 1653039404.527354 seconds ago (Net::HTTP::Persistent::Error)
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/protocol.rb:157:in `readuntil'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/protocol.rb:167:in `readline'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http/response.rb:40:in `read_status_line'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http/response.rb:29:in `read_new'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1494:in `block in transport_request'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1491:in `catch'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1491:in `transport_request'
migrations_1                                                |   from /usr/local/lib/ruby/2.5.0/net/http.rb:1464:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:964:in `block in request'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:662:in `connection_for'
migrations_1                                                |   from /usr/local/bundle/gems/net-http-persistent-3.1.0/lib/net/http/persistent.rb:958:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:696:in `request'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:344:in `response'
migrations_1                                                |   from /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:305:in `query'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:22:in `query'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:26:in `update'
migrations_1                                                |   from /usr/local/bundle/gems/mu-auth-sudo-0.1.0/lib/mu/auth-sudo.rb:8:in `block in included'
migrations_1                                                |   from /usr/src/app/ext/web.rb:129:in `batch_insert'
migrations_1                                                |   from /usr/src/app/ext/web.rb:78:in `execute!'
migrations_1                                                |   from /usr/src/app/ext/web.rb:151:in `block in execute_migrations'
migrations_1                                                |   from /usr/src/app/ext/web.rb:149:in `each'
migrations_1                                                |   from /usr/src/app/ext/web.rb:149:in `execute_migrations'
migrations_1                                                |   from /usr/src/app/ext/web.rb:214:in `boot'
migrations_1                                                |   from /usr/src/app/ext/web.rb:217:in `<top (required)>'
migrations_1                                                |   from web.rb:87:in `require_relative'
migrations_1                                                |   from web.rb:87:in `<main>'

claire-lovisa commented 2 years ago

I tested it as well and I face the same issue as Nordine, which I don't have when I use tenforce/virtuoso:1.3.2-virtuoso7.2.5.1.

nvdk commented 1 year ago

@claire-lovisa @nbittich could you retest with redpencil/virtuoso:1.1.0 ?

nbittich commented 1 year ago

@nvdk will try this week

claire-lovisa commented 1 year ago

@nvdk It's better, now the migration that used to fail comes through. But now an other migrations later down the line triggers this error (which doesn't happen if when using tenforce/virtuoso:1.3.2-virtuoso7.2.5.1):

migrations_1                                         | I, [2023-05-16T09:18:58.238544 #12]  INFO -- : Executing migration 20220602164623-fix-typo-in-position-name.sparql
migrations_1                                         | I, [2023-05-16T09:18:58.238660 #12]  INFO -- : Executing sudo query: DELETE DATA {
migrations_1                                         |   GRAPH <http://mu.semte.ch/graphs/organisatieportaal> {
migrations_1                                         |     <http://lblod.data.gift/concepts/fa4191f9d7050fe62ec3fc0e16541711> <http://www.w3.org/2004/02/skos/core#prefLabel> "Kapelaan van de kerken te Antwerpen en te Elsene (Geünifieerde anglikaanse kerk)" .
migrations_1                                         |   }
migrations_1                                         | }
migrations_1                                         | ;
migrations_1                                         | INSERT DATA {
migrations_1                                         |   GRAPH <http://mu.semte.ch/graphs/organisatieportaal> {
migrations_1                                         |     <http://lblod.data.gift/concepts/fa4191f9d7050fe62ec3fc0e16541711> <http://www.w3.org/2004/02/skos/core#prefLabel> "Kapelaan van de kerken te Antwerpen en te Elsene (Geünifieerde anglicaanse kerk)" .
migrations_1                                         |   }
migrations_1                                         | }
triplestore_1                                        | 09:18:58 ERRS_0 42000 SR186:SECURITY No permission to execute dpipe DB.DBA.L_O_LOOK_NE with user ID 107, group ID 107
migrations_1                                         | /usr/local/bundle/gems/sparql-client-3.0.1/lib/sparql/client.rb:351:in `block in response': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

redpencilio / docker-virtuoso

[regression]: migrations fail #7