search

jijijig / Albeobsa-Backend

알(뜰마)법사
https://albeobsa-front.vercel.app
MIT License
0 stars 1 forks source link

AWS EC2 환경에서 퀘사이존 크롤링 실패 #13

Closed chaiminwoo0223 closed 3 months ago

chaiminwoo0223 commented 3 months ago 
spring-boot-app  | 2024-05-22T13:25:51.825Z  WARN 1 --- [       Thread-2] org.openqa.selenium.os.ExternalProcess   : failed to copy the output of process 94
spring-boot-app  | 
spring-boot-app  | java.io.IOException: Stream closed
spring-boot-app  |  at java.base/java.io.BufferedInputStream.ensureOpen(BufferedInputStream.java:206) ~[na:na]
spring-boot-app  |  at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:411) ~[na:na]
spring-boot-app  |  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:405) ~[na:na]
spring-boot-app  |  at java.base/java.io.InputStream.transferTo(InputStream.java:796) ~[na:na]
spring-boot-app  |  at java.base/java.io.BufferedInputStream.implTransferTo(BufferedInputStream.java:657) ~[na:na]
spring-boot-app  |  at java.base/java.io.BufferedInputStream.transferTo(BufferedInputStream.java:637) ~[na:na]
spring-boot-app  |  at org.openqa.selenium.os.ExternalProcess$Builder.lambda$start$0(ExternalProcess.java:209) ~[selenium-os-4.14.1.jar!/:na]
spring-boot-app  |  at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
spring-boot-app  |
chaiminwoo0223 commented 3 months ago

아무래도 메모리 문제와 관련이 있는 것 같습니다. 빨리 해결하겠습니다.

chaiminwoo0223 commented 3 months ago 
spring-boot-app  | 퀘사이존
spring-boot-app  | 2024-05-29T12:51:36.062Z  INFO 1 --- [   scheduling-1] i.g.bonigarcia.wdm.WebDriverManager      : Using chromedriver 125.0.6422.78 (resolved driver for Chrome 125)
spring-boot-app  | 2024-05-29T12:51:36.070Z  INFO 1 --- [   scheduling-1] i.g.bonigarcia.wdm.WebDriverManager      : Exporting webdriver.chrome.driver as /root/.cache/selenium/chromedriver/linux64/125.0.6422.78/chromedriver
spring-boot-app  | 2024-05-29T12:51:40.462Z ERROR 1 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task
spring-boot-app  | 
spring-boot-app  | skhu.jijijig.exception.CrawlingProcessException: 크롤링 중 오류 발생: 데이터 추출 중 오류 발생: no such element: Unable to locate element: {"method":"css selector","selector":"div.user-nick-text"}
spring-boot-app  |   (Session info: chrome-headless-shell=125.0.6422.76)
spring-boot-app  | For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception
spring-boot-app  | Build info: version: '4.14.1', revision: '03f8ede370'
spring-boot-app  | System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.8.0-1008-aws', java.version: '21'
spring-boot-app  | Driver info: org.openqa.selenium.chrome.ChromeDriver
spring-boot-app  | Command: [207c05837c504eaaa83b5c6d93fb277e, findChildElement {id=f.1043271CBE781016A3F73B47ACF0483D.d.793924C2A884FEC29CAE374EA980E050.e.17, using=css selector, value=div.user-nick-text}]
spring-boot-app  | Capabilities {acceptInsecureCerts: false, browserName: chrome-headless-shell, browserVersion: 125.0.6422.76, chrome: {chromedriverVersion: 125.0.6422.78 (14db42ec38ad..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:38633}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:cdp: ws://localhost:38633/devtoo..., se:cdpVersion: 125.0.6422.76, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
spring-boot-app  | Session ID: 207c05837c504eaaa83b5c6d93fb277e
spring-boot-app  |  at skhu.jijijig.service.CrawlingService.crawlWebsite(CrawlingService.java:146) ~[!/:0.0.1-SNAPSHOT]
spring-boot-app  |  at skhu.jijijig.service.CrawlingService.performCrawlingForQuasarzone(CrawlingService.java:87) ~[!/:0.0.1-SNAPSHOT]
spring-boot-app  |  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
spring-boot-app  |  at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
spring-boot-app  |  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:385) ~[spring-tx-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at skhu.jijijig.service.CrawlingService$$SpringCGLIB$$0.performCrawlingForQuasarzone(<generated>) ~[!/:0.0.1-SNAPSHOT]
spring-boot-app  |  at skhu.jijijig.service.CrawlingService.scheduleCrawlingTasks(CrawlingService.java:45) ~[!/:0.0.1-SNAPSHOT]
spring-boot-app  |  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
spring-boot-app  |  at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
spring-boot-app  |  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:713) ~[spring-aop-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at skhu.jijijig.service.CrawlingService$$SpringCGLIB$$0.scheduleCrawlingTasks(<generated>) ~[!/:0.0.1-SNAPSHOT]
spring-boot-app  |  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
spring-boot-app  |  at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
spring-boot-app  |  at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) ~[spring-context-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at io.micrometer.observation.Observation.observe(Observation.java:499) ~[micrometer-observation-1.12.3.jar!/:1.12.3]
spring-boot-app  |  at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.1.4.jar!/:6.1.4]
spring-boot-app  |  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
spring-boot-app  |  at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[na:na]
spring-boot-app  |  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
spring-boot-app  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
spring-boot-app  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
spring-boot-app  |  at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
chaiminwoo0223 commented 3 months ago

name의 css selector가 문제였습니다...

chaiminwoo0223 commented 3 months ago

해결했습니다.^^