Memory Error

[Almars12345]

When i loade google model, i get "out of memory"issue . I try modifying the heap size from run configuration ,but it still not working.BTW,I'm using NetBeans to run the code below.

Here is the error

- > Exception in thread "main" java.lang.OutOfMemoryError: Cannot allocate new FloatPointer(900000000), totalBytes = 160602582, physicalBytes = 941273088
- >     at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:76)
- >     at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:445)
- >     at org.nd4j.linalg.api.buffer.FloatBuffer.<init>(FloatBuffer.java:57)
- >     at org.nd4j.linalg.api.buffer.factory.DefaultDataBufferFactory.createFloat(DefaultDataBufferFactory.java:238)
- >     at org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:1301)
- >     at org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:1276)
- >     at org.nd4j.linalg.api.ndarray.BaseNDArray.<init>(BaseNDArray.java:253)
- >     at org.nd4j.linalg.cpu.nativecpu.NDArray.<init>(NDArray.java:112)
- >     at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.create(CpuNDArrayFactory.java:248)
- >     at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4477)
- >     at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4439)
- >     at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:3692)
- >     at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readBinaryModel(WordVectorSerializer.java:238)
- >     at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadGoogleModel(WordVectorSerializer.java:134)
- >     at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadGoogleModel(WordVectorSerializer.java:111)
- >     at org.deeplearning4j.examples.nlp.word2vec.Word2VecRawTextExample.main(Word2VecRawTextExample.java:72)
- > Caused by: java.lang.OutOfMemoryError: Cannot allocate 160602582 + 3600000000 bytes (> Pointer.maxBytes)
- >     at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:543)
- >     at org.bytedeco.javacpp.Pointer.init(Pointer.java:121)
- >     at org.bytedeco.javacpp.FloatPointer.allocateArray(Native Method)
- >     at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:68)
- >     ... 15 more
- > ------------------------------------------------------------------------
- > BUILD FAILURE
- > ------------------------------------------------------------------------
- > Total time: 13.891 s
- > Finished at: 2016-11-30T22:46:15+03:00
- > Final Memory: 10M/155M
- > ------------------------------------------------------------------------
- > Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default-cli) on project dl4j-examples: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
- > 
- > To see the full stack trace of the errors, re-run Maven with the -e switch.
- > Re-run Maven using the -X switch to enable full debug logging.
- > 
- > For more information about the errors and possible solutions, please read the following articles:
- > [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
- > 

Here is my source 

public static void main(String[] args) throws Exception {

    // Gets Path to Text file
    String filePath = new ClassPathResource("raw_sentences.txt").getFile().getAbsolutePath();

    log.info("Load & Vectorize Sentences....");
    // Strip white space before and after for each line
    SentenceIterator iter = new BasicLineIterator(filePath);
    // Split on white spaces in the line to get words
    TokenizerFactory t = new DefaultTokenizerFactory();

    /*
        CommonPreprocessor will apply the following regex to each token: [\d\.:,"'\(\)\[\]|/?!;]+
        So, effectively all numbers, punctuation symbols and some special symbols are stripped off.
        Additionally it forces lower case for all tokens.
     */
    t.setTokenPreProcessor(new CommonPreprocessor());

    log.info("Building model....");
    Word2Vec vec = new Word2Vec.Builder()
            .minWordFrequency(5)
            .iterations(1)
            .layerSize(100)
            .seed(42)
            .windowSize(5)
            .iterate(iter)
            .tokenizerFactory(t)
            .build();

    log.info("Fitting Word2Vec model....");
    vec.fit();

    log.info("Writing word vectors to text file....");

    // Write word vectors to file
    WordVectorSerializer.writeWordVectors(vec, "pathToWriteto.txt");

    // Prints out the closest 10 words to "day". An example on what to do with these Word Vectors.
    log.info("Closest Words:");
    Collection<String> lst = vec.wordsNearest("day", 10);
    System.out.println("10 Words closest to 'day': " + lst);
       File gModel = new File("C:/Users/the king/Documents/NetBeansProjects/GoogleNews-vectors-negative300.bin.gz");

Word2Vec vec1 = WordVectorSerializer.loadGoogleModel(gModel, true);

Finally POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>dl4j-examples</artifactId>

    <parent>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-examples-parent</artifactId>
        <version>0.7-SNAPSHOT</version>
    </parent>

    <name>DeepLearning4j Examples</name>

    <repositories>
        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <distributionManagement>
        <snapshotRepository>
            <id>sonatype-nexus-snapshots</id>
            <name>Sonatype Nexus snapshot repository</name>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </snapshotRepository>
        <repository>
            <id>nexus-releases</id>
            <name>Nexus Release Repository</name>
            <url>http://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
        </repository>
    </distributionManagement>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.nd4j</groupId>
                <artifactId>nd4j-native-platform</artifactId>
                <version>${nd4j.version}</version>
            </dependency>
            <dependency>
                <groupId>org.nd4j</groupId>
                <artifactId>nd4j-cuda-7.5-platform</artifactId>
                <version>${nd4j.version}</version>
            </dependency>
            <dependency>
                <groupId>org.nd4j</groupId>
                <artifactId>nd4j-cuda-8.0-platform</artifactId>
                <version>${nd4j.version}</version>
            </dependency>
       </dependencies>
    </dependencyManagement>

    <dependencies>
        <!-- ND4J backend. You need one in every DL4J project. Normally define artifactId as either "nd4j-native-platform" or "nd4j-cuda-7.5-platform" -->
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.backend}</artifactId>
        </dependency>

        <!-- Core DL4J functionality -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nlp</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

        <!-- deeplearning4j-ui is used for HistogramIterationListener + visualization: see http://deeplearning4j.org/visualization -->
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui_2.10</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

        <!-- Force guava versions for using UI/HistogramIterationListener -->
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>${guava.version}</version>
        </dependency>

        <!-- datavec-data-codec: used only in video example for loading video data -->
        <dependency>
            <artifactId>datavec-data-codec</artifactId>
            <groupId>org.datavec</groupId>
            <version>${datavec.version}</version>
        </dependency>

        <!-- Used in the feedforward/classification/MLP* and feedforward/regression/RegressionMathFunctions example -->
        <dependency>
            <groupId>jfree</groupId>
            <artifactId>jfreechart</artifactId>
            <version>${jfreechart.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jfree</groupId>
            <artifactId>jcommon</artifactId>
            <version>${jcommon.version}</version>
        </dependency>

        <!-- Used for downloading data in some of the examples -->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.3.5</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>${exec-maven-plugin.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <executable>java</executable>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${maven-shade-plugin.version}</version>
                <configuration>
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                    <shadedClassifierName>${shadedClassifier}</shadedClassifierName>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>org/datanucleus/**</exclude>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Could you please help me solve this problem .

[saudet]

You'll need to increase the limit to about 40g of RAM to run that. Are you saying that you are getting this error with unmodified example code?

[Almars12345]

@saudent Yes getting this error with the above code.Are sure i need 40g? My laptop RAM is 8 G.I think 40 g too much .

[Almars12345]

@saudet

[saudet]

What about Word2VecSentimentRNN? Do you have any problems executing that one?

[raver119]

I'm not sure why 40gb would be needed there. It's pure oom during w2v google model load, which uses around 4gb itself for floats only + strings.

[Almars12345]

@saudet I executed the same code without loading the model, it works.However , loading the model gives me the error .Regarding to word2vecsentiment , it works .I believe the problem with loading the model.it consumes so much memory. I increased the heap size -Xmx6G , but still .Also, i run the same program with 16 RAM, the same issue appears.

[raver119]

Here's where google model loading happens: https://github.com/deeplearning4j/deeplearning4j/blob/6c11cd24ed47c37d535ec38ca7a7afbcb1b50891/deeplearning4j-nlp-parent/deeplearning4j-nlp/src/main/java/org/deeplearning4j/models/embeddings/loader/WordVectorSerializer.java#L224-L224

TL/DR: syn0 is created, which matches model dimensionality, in case of Google Model - it's 3m x 300. After it's created - vectors are read one by one, and inserted into syn0.

I don't see any real "extra" memory use here.

[raver119]

Btw, as fast workaround you could use w2v models with smaller dimensionality.

[Almars12345]

@raver119 so , what i have to do to fix it? And how can I decrease the dimensionality ?

[raver119]

I'm not sure at this moment, if there's anything to be fixed. I've pointed you to the code, so you can see it yourself.

As for dimensionality: just download another pre-trained model, or train your own. There's plenty of other models available for download in the web. Google Model isn't the only :)

I.e. here: https://github.com/3Top/word2vec-api

[Almars12345]

@raver119 i'm wondering if i can load GloVe model such as crawl, twitter..etc., using deeplearing4j library ? is there any sample i can have a look at thank you

[raver119]

Yes, just use the same methods. There's 2 widely used formats out there, and we support them both. One is binary model (like google model) and other one is csv. So you'll definitely be fine using WordVectorSerializer utility methods

[Almars12345]

@raver119 hi again , is it possible to run the word2vect code in java project , i mean not maven? and what are the libraries needed in order to do that.

[raver119]

We don't really support setups without build systems being used. Sure, you're free to go that way, but you'll be on your own there. I.e. parse maven dependency tree, or something like that.

[siddharthBohra]

Even i am faced the same issue like "Unable to allocate memory" for the Google-news model, then tried with CBOW model (words.cbow.s200.w2v.bin.gz). It reads the 1st word in words.cbow.s200.w2v.bin file and exit with error message like unable to read the file format.

I have RAM: 8GB, IDE: IntelliJ, xmx value: 1024M.

Can you please tell me, how to solve this error.?

[agibsonccc]

Yeah you need a larger machine and larger heap space. You need a bigger machine to run the google model. There's nothing for us to do here.