search

danielvarga / hunglish-webapp

Automatically exported from code.google.com/p/hunglish-webapp
0 stars 0 forks source link

scheduling of harness/indexing #43

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Most blokkolja a quartz a második harness futtatást. Feladat: schedule-oljon 
inkább.

Job-nak hivjuk azt, amikor egymás után lefut egy control_harness és egy 
indexelés.

A quartz-nak garantalnia kell, hogy egyszerre csak egy job fut. (Ha tobb fut, 
az annyira haza tudja vagni a konzisztenciat, hogy legjobb, ha a 
control_harness es az indexelo maguk is resen vannak, es vegeznek valamife'le 
lockolast.)

A quartz a kovetkezo trivialis modon job schedule-ol: Egy queue-ban gyujti a 
job request-eket. Ha eppen szabad a munkapad, akkor rateszi a sor elejen allo 
jobot.

Kik tudnak job requestet kuldeni? Egyreszet egy idozito, ami x percenkent 
(harnessCronPeriodMinutes=x property) megcsinalja ezt. Masreszt, ha a 
konfiguralaskor (instantHarnessSchedule=True property) ezt megengedjuk, akkor a 
fileUploadController is.

Megjegyzések:

1. Ha egynél több várakozó job request van, akkor azok jelenleg nyugodtan 
összeolvaszthatók eggyé, mert ezek most tulajdonságmentesek.

2. Távoli jövő: Később lehet, hogy minimum három munkadarab-fajtát 
fogunk elkülöníteni, úgyismint: control_harness, dumplumfilter, indexing. 
Ezekre ugyanennyi munkapad lesz, és lényegesen komplexebb scheduling. Elvileg 
mindegyik teljesen consumer-producer. A startup overheadjük egyetlen oka 
annak, hogy nem mindegy, hogy milyen sorrendben vannak meghívva.

Original issue reported on code.google.com by Varga.Da...@gmail.com on 1 Mar 2011 at 8:36

GoogleCodeExporter commented 9 years ago

Original comment by Varga.Da...@gmail.com on 1 Mar 2011 at 8:40

GoogleCodeExporter commented 9 years ago 
A queue-hoz egy megjegyzés: 
azt hiszem elég, ha a táblában vannak tárolva a feldolgozatlan upload-ok 
(is_processed='N'). 
És ha lefut a job, akkor megnézi, hogy vannak-e még feldolgozatlan upload-ok,
és ha vannak, akkor azokra is rámegy.

Original comment by bpgergo on 1 Mar 2011 at 8:49

GoogleCodeExporter commented 9 years ago

Original comment by attila.b...@gmail.com on 1 Mar 2011 at 11:39

GoogleCodeExporter commented 9 years ago 
Revision 352
- uploadjob.cronexpression in application.properties: cron expr for starting 
execute(..) in UploadJob.java in a scheduled way
- cleaned up Indexer.java a bit

Original comment by attila.b...@gmail.com on 2 Mar 2011 at 2:06

GoogleCodeExporter commented 9 years ago

Original comment by attila.b...@gmail.com on 2 Mar 2011 at 9:29

GoogleCodeExporter commented 9 years ago 
Harness should return an exit code of 1 only when it finished successfully.
The indexer should not start after Harness error.

Original comment by bpgergo on 3 Mar 2011 at 9:55

GoogleCodeExporter commented 9 years ago 
Revision 367

Original comment by attila.b...@gmail.com on 10 Mar 2011 at 9:35

GoogleCodeExporter commented 9 years ago 
Revision 375
a mostani verzióval az van, hogy egy jobot sem lehet elindítani a 
felületről, mert az időzített jobbal ütközik, így-e:

2011-03-13 20:55:16,022 [http-8080-4] INFO  
hu.mokk.hunglish.web.UploadController - upload saved in db. starting upload job 
...
2011-03-13 20:55:16,022 [http-8080-4] INFO  
hu.mokk.hunglish.web.UploadController - triggered job manually.
2011-03-13 20:55:16,023 [http-8080-4] ERROR 
hu.mokk.hunglish.web.UploadController - Error while creating upload
org.quartz.ObjectAlreadyExistsException: Unable to store Job with name: 
'jobDetail' and group: 'DEFAULT', because one already exists with this 
identification.
    at org.quartz.simpl.RAMJobStore.storeJob(RAMJobStore.java:221)
    at org.quartz.simpl.RAMJobStore.storeJobAndTrigger(RAMJobStore.java:193)
    at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:798)
    at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243)
    at hu.mokk.hunglish.web.UploadController.startUploadJob(UploadController.java:79)
    at hu.mokk.hunglish.web.UploadController.create(UploadController.java:152)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.doInvokeMethod(HandlerMethodInvoker.java:710)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:167)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:414)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:402)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:771)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:563)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
    at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:195)
    at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:159)
    at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
    at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:90)
    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:417)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:71)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:343)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:177)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:188)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:355)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:149)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:113)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:636)
Hibernate: delete from upload where id=? and version=?

Original comment by bpgergo on 13 Mar 2011 at 8:01

GoogleCodeExporter commented 9 years ago 
Revision 376

Original comment by attila.b...@gmail.com on 17 Mar 2011 at 2:20

GoogleCodeExporter commented 9 years ago

Original comment by bpgergo on 18 Mar 2011 at 12:22