search

theotherp / nzbhydra2

Usenet meta search
Other
1.23k stars 75 forks source link

Settings/database migration from 0.2.231 fails #10

Closed whispy closed 6 years ago

whispy commented 6 years ago

Steps:

  1. Install Docker version of NZBHydra2 and start it — it starts with no issues.
  2. Navigate to Docker version of NZBHydra 1 and copy the settings.cfg and nzbhydra.db into a sub-folder of NZBHydra2's /data volume.
  3. Open NZBHydra2's web UI and navigate to System > Control > Migrate from NZBHydra 1.
  4. Select NZBHydra 1 is running on another machine or in a container.
  5. Enter locations of the files (see screenshot: https://vgy.me/1BebGq.png) and click Start Migration.
  6. Migration runs.

Results:

Note: I have not yet tried running with --xmx 512M, as Synology's Docker UI makes this difficult and my log files make it seem like this is not the issue. However, if you request, I will figure out how to run it with this argument.

Log (debug output enabled and indexer names redacted):

2018-01-06 16:14:44.910  WARN --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Deleting all indexers, indexer searches, searches, downloads and API accesses from database
2018-01-06 16:14:45.407  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Starting database migration
2018-01-06 16:14:45.665  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating 13 indexers from old database
2018-01-06 16:14:45.700 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.515 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.517 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.520 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.522 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.524 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.527 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.529 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.530 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.532 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.535 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.537 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.538 DEBUG --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating indexer IndexerEntity{id=0, name=REDACTED}
2018-01-06 16:14:46.540  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Successfully migrated indexers from old database
2018-01-06 16:14:46.657  INFO --- [0.0-5076-exec-5] org.nzbhydra.migration.SqliteMigration   : [IP: 172.17.0.1] Migrating 215 searches from old database
2018-01-06 16:14:46.818 ERROR --- [0.0-5076-exec-5] o.n.migration.FromPythonMigration        : [IP: 172.17.0.1] Error while migrating database

java.sql.SQLException: no such column: 'author'
    at org.sqlite.jdbc3.JDBC3ResultSet.findColumn(JDBC3ResultSet.java:48)
    at org.sqlite.jdbc3.JDBC3ResultSet.getString(JDBC3ResultSet.java:443)
    at org.nzbhydra.migration.SqliteMigration.migrateSearches(SqliteMigration.java:278)
    at org.nzbhydra.migration.SqliteMigration.migrate(SqliteMigration.java:110)
    at org.nzbhydra.migration.SqliteMigration.migrate(SqliteMigration.java:100)
    at org.nzbhydra.migration.SqliteMigration$$FastClassBySpringCGLIB$$72e2874a.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at org.nzbhydra.migration.SqliteMigration$$EnhancerBySpringCGLIB$$7dae4580.migrate(<generated>)
    at org.nzbhydra.migration.FromPythonMigration.startMigration(FromPythonMigration.java:103)
    at org.nzbhydra.migration.FromPythonMigration.migrateFromFiles(FromPythonMigration.java:65)
    at org.nzbhydra.migration.FromPythonMigration$$FastClassBySpringCGLIB$$25126370.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
    at org.nzbhydra.migration.FromPythonMigration$$EnhancerBySpringCGLIB$$7c1da1b2.migrateFromFiles(<generated>)
    at org.nzbhydra.migration.MigrationWeb.migrateFromFiles(MigrationWeb.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.nzbhydra.auth.HydraAnonymousAuthenticationFilter.doFilter(HydraAnonymousAuthenticationFilter.java:98)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1457)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

Please let me know if I can provide any more info or testing. Thanks for putting all this effort into the rewrite!

theotherp commented 6 years ago

Thanks for the report. I'm getting feedback left and right, so it might take a bit until I can take a look.

whispy commented 6 years ago

I'm running NZBHydra 1, version 0.2.231 (see: https://vgy.me/pEfIX2.png).

theotherp commented 6 years ago

Yeah sorry, I didn't see that in the headline ;-)

theotherp commented 6 years ago

There's a column missing in the database that was added with 0.2.151. I can't really explain why it's not there. That should not be possible. That could mean that there are any number of possible database states out there. Oh my god.

I'll have to release a new update for v1 which makes sure everything is expected but that will take time. Considering how many reports I'm getting for v2 you're better off waiting a bit ;-)

whispy commented 6 years ago

Okay, thanks. Is that also why the settings.cfg import fails? I can do without my old database, but manually recreating all the settings and indexers in the UI would be a bit of a pain (but doable)

EDIT: I can send you my DB file too, if you'd like.

theotherp commented 6 years ago

Please post your debug infos. I didn't really expect so many (or any issues) with the migration.

The settings should be migrated even if the database migration fails.

theotherp commented 6 years ago

Please post your debug infos from v2 (http://127.0.0.1:5076/system/bugreport)

whispy commented 6 years ago

Debuginfos: https://dl.dropboxusercontent.com/s/sm8ckzmtvys6ya3/nzbhydra-debuginfos-2018-01-06-16-26.zip

theotherp commented 6 years ago

I'm seeing

Error while reading old settings file
java.io.IOException: Is a directory

Make sure that the settings file you provided is correct.

whispy commented 6 years ago

That was from a previous attempt where I just entered /data/nzbhydra-old-migration into the UI to see if it needed the filename or not. Other attempts in the debuginfos should have the full file paths.

EDIT: And I was doing the manual migration, as my V1 was running in a Docker container.

theotherp commented 6 years ago

Hm shit, the migration will fail when the database migration doesn't work.

As a workaround you could shut down your v1 instance, move the database files and start v1 again. Then try to migrate again. The v1 database should be reinitialized and match the expectations of the migration process.

theotherp commented 6 years ago

I really want to know what your database looks like and why it looks like that. The missing columns were added 2016 and Hydra shouldn't even work without them...

whispy commented 6 years ago

I figured it out, and it was me being silly — apologies for wasting your time.

I noticed when moving the database files, as you wrote earlier, that I was mistakenly using an older DB file from the end of 2016 (probably left over from some previous troubleshooting) that was in the top-level of my /hydra folder, and not the most up-to-date database. I corrected this and used the most recent DB file, and the migration worked fine.

Sorry for any potential stress! Keep up the awesome work!

whispy commented 6 years ago

Sent you a small donation to apologize :P

theotherp commented 6 years ago

Aaargh :-D

Well, I'm really glad that you figured it out quickly. I'll add code that checks the database version in the database file directly.

Thanks for the donation.