Migrate Mongo DB PROD data

RobertCzaja commented 2 months ago

From MongoDB dev to prod - prepare instruction step by step & CLI Move that DEV data to MongoDB prod Data to have everything in one place. At the very beginning I need to prepare some script that on demand can move data from Mongo DB to Postgres (because it will be useful also in migration 4developers data)

[x] WONT DO: unit test that method: pl.api.itoffers.provider.justjoinit.service.JustJoinItPayloadExtractor#convert
[x] group it by slug (in order to save in NoSQL the newest one)
[x] check if it isn't already saved in MongoDB
[x] add limit
[x] shows stats
[x] move FixReport & CliFixParam to some shared module

[x] migrate MongoDB offers (ScrappingId: 345c1efb-0223-43dc-b9c3-26b43d61128e)

INFO [12:59:42] p.a.i.p.j.u.c.ImportJJITOffersFromFileCli - Script finished successfully: 
fetchedCount: 2101
limit: 9999999
processingStarted: 1211
migrated: 1211
all offers in s3 file: 21536

Finally there is 25942 Offers in MongoDB

[ ] create next cli to migrate JJIT MongoDB Offers to Postgres Offer

RobertCzaja commented 1 week ago

java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
        at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
        at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
        at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
        at pl.api.itoffers.shared.aws.AwsS3Connector.fetchJson(AwsS3Connector.java:34)
        at pl.api.itoffers.provider.justjoinit.ui.cli.ImportJJITOffersFromFileCli.saveInMongoDBJJITOffersStoredInS3(ImportJJITOffersFromFileCli.java:47)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
        at pl.api.itoffers.provider.justjoinit.ui.cli.ImportJJITOffersFromFileCli$$SpringCGLIB$$0.saveInMongoDBJJITOffersStoredInS3(<generated>)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.shell.command.invocation.InvocableShellMethod.doInvoke(InvocableShellMethod.java:306)
        at org.springframework.shell.command.invocation.InvocableShellMethod.invoke(InvocableShellMethod.java:232)
        at org.springframework.shell.command.CommandExecution$DefaultCommandExecution.evaluate(CommandExecution.java:230)
        at org.springframework.shell.Shell.evaluate(Shell.java:248)
        at org.springframework.shell.Shell.run(Shell.java:159)
        at org.springframework.shell.jline.InteractiveShellRunner.run(InteractiveShellRunner.java:72)
        at org.springframework.shell.DefaultShellApplicationRunner.run(DefaultShellApplicationRunner.java:66)

Solutions: https://stackoverflow.com/questions/19108549/stringbuilder-append-cause-out-of-memory https://coderanch.com/t/626258/java/java-lang-OutOfMemoryError-Java-heap

try increasing the JVM heap memory (-Xms, -Xmx).
something like this would increase max memory to 8 GB if you have it 'java -Xmx8G -cp PATH_TO_CLASSES com.yourpackage.YourJSONObject

LOCAL:
 INFO [09:02:42] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 75 MB; free: 44 MB; allocated: 120 MB; max: 4096 MB: context: before download
 INFO [09:02:42] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 102 MB; free: 17 MB; allocated: 120 MB; max: 4096 MB: context: after download
 INFO [09:03:00] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 368 MB; free: 17 MB; allocated: 386 MB; max: 4096 MB: context: after string builder
 INFO [09:03:00] p.a.i.s.utils.monitor.MemoryUsage - [memory-usage] used: 368 MB; free: 17 MB; allocated: 386 MB; max: 4096 MB: context: after Reader closing

PROD:
 INFO [07:28:48] p.a.i.shared.aws.AwsS3Connector - [memory-usage] used: 56 MB; free: 19 MB; allocated: 76 MB; max: 220 MB: context: before download
 INFO [07:28:49] p.a.i.shared.aws.AwsS3Connector - [memory-usage] used: 66 MB; free: 9 MB; allocated: 76 MB; max: 220 MB: context: after download
 INFO [07:28:52] o.s.b.a.l.ConditionEvaluationReportLogger - 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
ERROR [07:28:52] o.s.boot.SpringApplication - Application run failed
java.lang.OutOfMemoryError: Java heap space

https://instances.vantage.sh/aws/ec2/t3.micro

failed to register layer: write /it-offers-0.0.1-SNAPSHOT.jar: no space left on device

Solution: perform migration on Local with MongoDB Prod

RobertCzaja commented 1 week ago

Migrated!

RobertCzaja / it-offers-api

Migrate Mongo DB PROD data #18