Closed RobertCzaja closed 1 week ago
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
at pl.api.itoffers.shared.aws.AwsS3Connector.fetchJson(AwsS3Connector.java:34)
at pl.api.itoffers.provider.justjoinit.ui.cli.ImportJJITOffersFromFileCli.saveInMongoDBJJITOffersStoredInS3(ImportJJITOffersFromFileCli.java:47)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
at pl.api.itoffers.provider.justjoinit.ui.cli.ImportJJITOffersFromFileCli$$SpringCGLIB$$0.saveInMongoDBJJITOffersStoredInS3(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:230)
at org.springframework.shell.Shell.evaluate(Shell.java:248)
at org.springframework.shell.Shell.run(Shell.java:159)
at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:72)
at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:66)
Solutions: https://stackoverflow.com/questions/19108549/stringbuilder-append-cause-out-of-memory https://coderanch.com/t/626258/java/java-lang-OutOfMemoryError-Java-heap
LOCAL:
INFO [09:02:42] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 75 MB; free: 44 MB; allocated: 120 MB; max: 4096 MB: context: before download
INFO [09:02:42] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 102 MB; free: 17 MB; allocated: 120 MB; max: 4096 MB: context: after download
INFO [09:03:00] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 368 MB; free: 17 MB; allocated: 386 MB; max: 4096 MB: context: after string builder
INFO [09:03:00] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 368 MB; free: 17 MB; allocated: 386 MB; max: 4096 MB: context: after Reader closing
PROD:
INFO [07:28:48] p.a.i.shared.aws.AwsS3Connector - [memory-usage] used: 56 MB; free: 19 MB; allocated: 76 MB; max: 220 MB: context: before download
INFO [07:28:49] p.a.i.shared.aws.AwsS3Connector - [memory-usage] used: 66 MB; free: 9 MB; allocated: 76 MB; max: 220 MB: context: after download
INFO [07:28:52] o.s.b.a.l.ConditionEvaluationReportLogger -
Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
ERROR [07:28:52] o.s.boot.SpringApplication - Application run failed
java.lang.OutOfMemoryError: Java heap space
https://instances.vantage.sh/aws/ec2/t3.micro
failed to register layer: write /it-offers-0.0.1-SNAPSHOT.jar: no space left on device
Solution: perform migration on Local with MongoDB Prod
Migrated!
From MongoDB dev to prod - prepare instruction step by step & CLI Move that DEV data to MongoDB prod Data to have everything in one place. At the very beginning I need to prepare some script that on demand can move data from Mongo DB to Postgres (because it will be useful also in migration 4developers data)
[x] WONT DO: unit test that method:
pl.api.itoffers.provider.justjoinit.service.JustJoinItPayloadExtractor#convert
[x] group it by slug (in order to save in NoSQL the newest one)
[x] check if it isn't already saved in MongoDB
[x] add limit
[x] shows stats
[x] move
FixReport
&CliFixParam
to some shared module[x] migrate MongoDB offers (ScrappingId:
345c1efb-0223-43dc-b9c3-26b43d61128e
)Finally there is 25942 Offers in MongoDB
[ ] create next cli to migrate JJIT MongoDB Offers to Postgres Offer