search

popbr / data-integration

Apache License 2.0
1 stars 4 forks source link

Fix project's exception #6

Closed aubertc closed 1 year ago

aubertc commented 1 year ago

As of now, I cannot execute the project:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project Database-IO: An exception occured while executing the Java class. Cannot read the array length because "fileN" is null -> [Help 1]

Can you investigate?

aubertc commented 1 year ago

I still face this issue. You too, @MNSleeper ?

MNSleeper commented 1 year ago

I haven't seen that last I executed. Might have been something to do with the methods I commented out, though.

aubertc commented 1 year ago

I still have it:

mvn exec:java -Dexec.mainClass="com.mycompany.app.App"
[INFO] Scanning for projects...
[INFO] 
[INFO] -------------------< com.mycompany.app:Database-IO >--------------------
[INFO] Building Database-IO 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ Database-IO ---

[WARNING] 
java.lang.NullPointerException: Cannot read the array length because "fileN" is null
    at com.mycompany.app.App.EstablishFileList (App.java:156)
    at com.mycompany.app.App.main (App.java:68)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:833)
[WARNING] thread Thread[mysql-cj-abandoned-connection-cleanup,5,com.mycompany.app.App] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[mysql-cj-abandoned-connection-cleanup,5,com.mycompany.app.App] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.mycompany.app.App,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:803)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:293)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.562 s
[INFO] Finished at: 2022-10-20T13:54:01-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project Database-IO: An exception occured while executing the Java class. Cannot read the array length because "fileN" is null -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

this is on a fresh copy of the project. I would assume it occurs because of https://github.com/popbr/data-integration/blob/4c50145d8bd150d391f6b8f4e93191cdb2d3fb55/Project/Database-IO/src/main/java/com/mycompany/app/App.java#L68

since there is nothing in the downloads folder.

MNSleeper commented 1 year ago

So should that line be commented out, should we put a dummy db in the downloads folder, or should something else be done?

aubertc commented 1 year ago

Use a conditional: if there are databases in that folder, process them, otherwise, skip.

MNSleeper commented 1 year ago

Alright, thank you for the methodology

MNSleeper commented 1 year ago

I have implemented your suggestion with a try catch, but the errors in git are still present, and I can't make heads or tails of the result. On my end for compiling/executing, I'm not running into any errors.

aubertc commented 1 year ago

You should start with fresh copies of your project to see the kind of issues github or I are facing.

My latest error is

mvn exec:java -Dexec.mainClass="com.mycompany.app.App"
[INFO] Scanning for projects...
[INFO] 
[INFO] -------------------< com.mycompany.app:Database-IO >--------------------
[INFO] Building Database-IO 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ Database-IO ---

[WARNING] 
java.io.FileNotFoundException: /home/caubert/travail/git/data-integration/Project/Database-IO/target/DBInfo.xml (No such file or directory)
    at java.io.FileInputStream.open0 (Native Method)
    at java.io.FileInputStream.open (FileInputStream.java:216)
    at java.io.FileInputStream.<init> (FileInputStream.java:157)
    at java.io.FileInputStream.<init> (FileInputStream.java:111)
    at sun.net.www.protocol.file.FileURLConnection.connect (FileURLConnection.java:86)
    at sun.net.www.protocol.file.FileURLConnection.getInputStream (FileURLConnection.java:189)
    at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity (Unknown Source)
    at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion (Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse (Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse (Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse (Unknown Source)
    at org.apache.xerces.parsers.DOMParser.parse (Unknown Source)
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse (Unknown Source)
    at javax.xml.parsers.DocumentBuilder.parse (DocumentBuilder.java:206)
    at com.mycompany.app.App.GetURLList (App.java:584)
    at com.mycompany.app.App.CreateURLList (App.java:191)
    at com.mycompany.app.App.main (App.java:73)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:833)
[WARNING] thread Thread[mysql-cj-abandoned-connection-cleanup,5,com.mycompany.app.App] was interrupted but is still alive after waiting at least 15000msecs
[WARNING] thread Thread[mysql-cj-abandoned-connection-cleanup,5,com.mycompany.app.App] will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=com.mycompany.app.App,maxpri=10]
java.lang.IllegalThreadStateException
    at java.lang.ThreadGroup.destroy (ThreadGroup.java:803)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:293)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.970 s
[INFO] Finished at: 2022-10-21T10:51:48-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project Database-IO: An exception occured while executing the Java class. /home/caubert/travail/git/data-integration/Project/Database-IO/target/DBInfo.xml (No such file or directory) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

The first error is about the file data-integration/Project/Database-IO/target/DBInfo.xml missing.

MNSleeper commented 1 year ago

It might be that the .gitignore prevents that from being uploaded. I'll add an exception and see if that fixes it.

aubertc commented 1 year ago

Ok, you should try to use the bast practices mentioned here.

MNSleeper commented 1 year ago

Wouldn't adding a rule that states to not ignore .xml files in the "target" folder work as well?

aubertc commented 1 year ago

There are multiple ways to achieve this, indeed, I'm not sure it matters too much.

aubertc commented 1 year ago

This issue is still present:

java.io.FileNotFoundException: /home/caubert/travail/git/data-integration/Project/Database-IO/target/DBInfo.xml (No such file or directory)

and prevents me from experimenting, as you suggest here.

MNSleeper commented 1 year ago

I added a sample DBInfo file that should remedy this.

Otherwise, Lines 72 & 73:

72 String URLPath = BasePath + File.separator + "target" + File.separator + "DBInfo.xml"; 73 String[] URLList = CreateURLList(URLPath);

can be commented out to bypass all uses of anything related to DBInfo.xml

aubertc commented 1 year ago

Ok, I can now compile and execute the code, but now I have another question:

[INFO] Scanning for projects...
[INFO] 
[INFO] -------------------------< popbr:Database-IO >--------------------------
[INFO] Building Database-IO 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:3.0.0:java (default-cli) @ Database-IO ---

There are no databases in the downloads folder.
Please download at least one database before running the program again.
Finished

Is the target/DBInfo.xml file simply ignored?

MNSleeper commented 1 year ago

Now that I think about it, the program scans the DBInfo file to scrape sites, then puts those scraped files in the DB. It's done to allow Users to put DBs directly in the download without needing them to specify websites to search. So, DBInfo is not needed only when users put files directly in the folder

MNSleeper commented 1 year ago

I can send the sample database I use.

aubertc commented 1 year ago

Now that I think about it, the program scans the DBInfo file to scrape sites, then puts those scraped files in the DB. It's done to allow Users to put DBs directly in the download without needing them to specify websites to search. So, DBInfo is not needed only when users put files directly in the folder

So, what is the protocol exactly? You can either drop files in the DB folder, or supply the information in the xml file, is that the idea? Please clarify.

MNSleeper commented 1 year ago

The idea is that a user can

  1. Specify a site to scrape databases from, which will then be put into the downloads folder

and/or

  1. Put databases into the downloads folder themselves, without specifying a site to scrape

Once the program starts, it goes through the scraping process first to get any sites the user specifies. With that done, it then goes to scan every Database in the downloads folder and parse them. The user doesn't have to both specify a site to scrap and manually put databases into the folder, but they must do one. Otherwise, there are not databases, and the program has nothing to parse.

aubertc commented 1 year ago

I don't think this mechanism is working as intended. I do have the Project/Database-IO/target/DBInfo.xml containing

<Databases>
    <DB id= "NSF">
        <Name>NSF Award Search</Name>
        <URL>https://www.nsf.gov/awardsearch/download.jsp</URL>
    </DB>   
</Databases>

(as in the repo), but my downloads folder only contains Data Spreadsheet.xlsx and I still have the error mentioned earlier.

MNSleeper commented 1 year ago

Interesting. It might be worth noting that the DBInfo.xml isn't doing anything right now since the scraping method isn't implemented. I have no idea as to why your execution doesn't recognize the database in the downloads folder, though.

aubertc commented 1 year ago

It might be worth noting that the DBInfo.xml isn't doing anything right now since the scraping method isn't implemented

I was not aware of that.

I have no idea as to why your execution doesn't recognize the database in the downloads folder, though.

Can you make a fresh copy of the repo (=clone it), run it, and let me know if you see the same error?

MNSleeper commented 1 year ago

I made a fresh clone and ran it. It executed perfectly for the Database that gets downloaded from the repo. Parsing, Inputting, and referencing all went smoothly. The only problems I ran into were database privileges, but that's because I forgot to change the LoginInfo.xml stored Username and Password for SQL.

aubertc commented 1 year ago

Ok, I'll try again.

aubertc commented 1 year ago

This is fixed with https://github.com/popbr/data-integration/commit/b173713e3c36255a64e0523245faaafa4b14bc3d