search

mifmax / vosao

Automatically exported from code.google.com/p/vosao
GNU Lesser General Public License v2.1
0 stars 0 forks source link

Speed up import/export. #238

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Current import/export takes a lot of time because of the way how it's 
organized. It uses chain of subsequent 25sec App Engine tasks.

The reason for this is found deeply in data model where relations are organized 
using surrogate keys unique for every app deployment.

To improve speed of these operations we need to do them in parallel tasks.

Unfortunately tree like structure with surrogate key relations are not possible 
to make parallel because child-parent key generation dependency.

To change this situation and open way for parallel processing we need to 
replace surrogate key relations to primary natural key relations.

To do:
1. Replace surrogate keys to natural keys for all relations.
2. Change import/export to use parallel tasks.

Original issue reported on code.google.com by kinyelo@gmail.com on 1 Jul 2010 at 11:47

GoogleCodeExporter commented 8 years ago 
But datastore get by key (surrogate key) operation is the fastest way load data.

Original comment by kinyelo@gmail.com on 1 Jul 2010 at 12:18

GoogleCodeExporter commented 8 years ago 
The most numerous items in import/export files are pages and files therefore we 
need to parallel operations on these types. PageEntity already doesn't have 
surrogate key relations so we need to change only FolderEntity and FileEntity.

Original comment by kinyelo@gmail.com on 1 Jul 2010 at 12:53

GoogleCodeExporter commented 8 years ago 
While developing parallel import I hit GAE "simultaneous dynamic request limit" 
which prevents some tasks from execution. 

This limit actually makes parallel import/export unreliable and buggy because 
some of import tasks are been simply dropped by GAE.

I will leave this issue open for future development.

Original comment by kinyelo@gmail.com on 11 Jul 2010 at 4:10

GoogleCodeExporter commented 8 years ago 
Hi Alex,

I can't get import to work properly. I think the reasons you put above are 
probably the case.

It seems to import the page structure (in the file resources) but doesn't add 
them to the content tree.

Also, sometimes it won't process all of the page structure (only will have a 
few items in the file resources).

What about writing some kind of java applet or flash app or something that 
sends files individually to the server rather than using the server to do it????

Cheers,

Roland.

Original comment by rquast%f...@gtempaccount.com on 29 Jul 2010 at 5:12

GoogleCodeExporter commented 8 years ago 
Just an update to the last comment... I got the import to work. I tried it 
again (at a later time) and it worked!

Original comment by rquast%e...@gtempaccount.com on 29 Jul 2010 at 11:57

GoogleCodeExporter commented 8 years ago 
I can't get import to work properly too.
It seems to import the page structure (in the file resources) but doesn't add 
them to the content tree.
I tried serveral times. 
How roland solve this problem? maybe I should try more?

Original comment by akand...@gmail.com on 22 Nov 2010 at 8:13

GoogleCodeExporter commented 8 years ago 
Please invite support@vosao.org as developer and send user admin 
account/password to support@vosao.org

Original comment by kinyelo@gmail.com on 3 Dec 2010 at 9:08

GoogleCodeExporter commented 8 years ago 
When you write about parent/child keys, are you talking about Entity Groups? I 
haven't seen yet in Vosao a relationship of that type. Everything extends a 
VosaoEntity that has a surrogate key, and if that's the case parallel 
operations should be possible.
Am I missing something?
Anyway, I like references using natural keys cause they make it easy to 
traverse entities when browsing the datastore with the admin console.

Original comment by theb...@emanueleziglioli.it on 30 Nov 2011 at 8:13

GoogleCodeExporter commented 8 years ago 
I got same result. Whem I use my localhost app engine server then export take 
about 10 second. But when I use app engine server hosting on google servers 
then it is not possible to do export. I try to wait about 3000 second and still 
no export proceed.
This issue probably need more investigation.

Original comment by jaromir....@aszol.cz on 4 Jul 2012 at 9:31

GoogleCodeExporter commented 8 years ago 
Hi maybee I found the cause of why export is not possible to do here are many 
examples.
Login to https://appengine.google.com/ and check your log and try to search for 
any errors like this:
2012-07-12 22:16:04.259

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [ERROR] 
AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz

W 2012-07-12 22:16:04.270

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [ERROR] 
AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz.txt

W 2012-07-12 22:16:04.277

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [INFO] 
AbstractSubscriber - -Export folder null 0

W 2012-07-12 22:16:07.692

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [INFO] 
AbstractSubscriber - -Export finished. 197

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: 157058 [Request 
FD10774F] INFO org.jabsorb.reflect.ClassAnalyzer - analyzing 
org.vosao.service.back.impl.ConfigServiceImpl

2012-07-12 22:11:54.633

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [ERROR] 
AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz

W 2012-07-12 22:11:54.642

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [ERROR] 
AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz.txt

W 2012-07-12 22:11:54.645

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [INFO] 
AbstractSubscriber - -Export folder null 0

W 2012-07-12 22:11:57.344

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: [ERROR] 
AbstractSubscriber - -java.util.zip.ZipException: duplicate entry: 
page/vyhody-pro-cleny/volani-za-vyhodne-ceny/ duplicate entry: 
page/vyhody-pro-cleny/volani-za-vyhodne-ceny/
java.util.zip.ZipException: duplicate entry: 
page/vyhody-pro-cleny/volani-za-vyhodne-ceny/
    at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:192)
    at org.vosao.business.impl.imex.task.ZipOutStreamTaskAdapterImpl.putNextEntry(ZipOutStreamTaskAdapterImpl.java:127)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addFolder(ResourceExporterImpl.java:73)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:95)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:98)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:98)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:98)
    at org.vosao.business.impl.ImportExportBusinessImpl.createExportFile(ImportExportBusinessImpl.java:180)
    at org.vosao.business.impl.mq.subscriber.ExportTaskSubscriber.onMessage(ExportTaskSubscriber.java:96)
    at org.vosao.business.impl.mq.MessageQueueImpl.execute(MessageQueueImpl.java:159)
    at org.vosao.servlet.MessageQueueServlet.execute(MessageQueueServlet.java:67)
    at org.vosao.servlet.MessageQueueServlet.doPost(MessageQueueServlet.java:48)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at org.vosao.filter.SiteFilter.doFilter(SiteFilter.java:107)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.RewriteFilter.doFilter(RewriteFilter.java:79)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:83)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.PluginCronFilter.doFilter(PluginCronFilter.java:78)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.LanguageFilter.doFilter(LanguageFilter.java:66)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.UpdateFilter.doFilter(UpdateFilter.java:78)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.InitFilter.doFilter(InitFilter.java:80)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.ContextFilter.doFilter(ContextFilter.java:74)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:102)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:477)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455)
    at com.google.tracing.TraceContext.runInContext(TraceContext.java:695)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
    at java.lang.Thread.run(Thread.java:679)

W 2012-07-12 22:11:57.344

[s~aszolweb/vosaocms-0-9.359765117814453489].<stderr>: 

[ERROR] AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz
[ERROR] AbstractBusinessImpl - -File not found. /tmp/exportFolder.vz.txt
[INFO] AbstractSubscriber - -Export folder null 0
[ERROR] AbstractSubscriber - -java.util.zip.ZipException: duplicate entry: 
tmp/index_en.bin duplicate entry: tmp/index_en.bin
java.util.zip.ZipException: duplicate entry: tmp/index_en.bin
    at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:175)
    at org.vosao.business.impl.imex.task.ZipOutStreamTaskAdapterImpl.putNextEntry(ZipOutStreamTaskAdapterImpl.java:127)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:126)
    at org.vosao.business.impl.imex.ResourceExporterImpl.addResourcesFromFolder(ResourceExporterImpl.java:98)
    at org.vosao.business.impl.ImportExportBusinessImpl.createExportFile(ImportExportBusinessImpl.java:180)
    at org.vosao.business.impl.mq.subscriber.ExportTaskSubscriber.onMessage(ExportTaskSubscriber.java:96)
    at org.vosao.business.impl.mq.MessageQueueImpl.execute(MessageQueueImpl.java:159)
    at org.vosao.servlet.MessageQueueServlet.execute(MessageQueueServlet.java:67)
    at org.vosao.servlet.MessageQueueServlet.doPost(MessageQueueServlet.java:48)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at org.vosao.filter.SiteFilter.doFilter(SiteFilter.java:106)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.RewriteFilter.doFilter(RewriteFilter.java:79)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:83)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.PluginCronFilter.doFilter(PluginCronFilter.java:78)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.LanguageFilter.doFilter(LanguageFilter.java:66)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.UpdateFilter.doFilter(UpdateFilter.java:78)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.InitFilter.doFilter(InitFilter.java:80)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.vosao.filter.ContextFilter.doFilter(ContextFilter.java:74)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:35)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:60)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.BackendServersFilter.doFilter(BackendServersFilter.java:97)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:94)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:370)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Original comment by jaromir....@aszol.cz on 12 Jul 2012 at 8:20

GoogleCodeExporter commented 8 years ago 
Please look at hte mentioned folder :

 duplicate entry: page/vyhody-pro-cleny/volani-za-vyhodne-ceny/ duplicate entry: page/vyhody-pro-cleny/volani-za-vyhodne-ceny/

Original comment by kinyelo@gmail.com on 12 Jul 2012 at 9:44

GoogleCodeExporter commented 8 years ago 
Yes I know about (from logs that I paste here) but I dont understand from where 
this duplicity come from (what can cause this). I search and here is described 
same problem. http://code.google.com/p/vosao/issues/detail?id=449&q=duplicate

Original comment by jaromir....@aszol.cz on 12 Jul 2012 at 10:08