search

kriegaex / Galileo-Openbook-Cleaner

HTML cleaner for Galileo Computing's openbooks, implemented in Java
Other
64 stars 6 forks source link

New openbook VCSHARP_2012 and proxy problems #10

Closed ifix01 closed 11 years ago

ifix01 commented 11 years ago

there is a new openbook available at galileo-openbooks: Visual C# 2012. This however is not yet known to the openbook cleaner, so i can't download and clean. Btw would be great if cleaning could be done independant from downloading so we don't need an update of the software for new books but could download them manually and use the tool just for cleaning up the HTML code.

kriegaex commented 11 years ago

Thanks for the notification. I do not continually check for new books. What you could have done: download the book manually, rename the archive to an existing one with similar HTML content structure (e.g. vcsharp_2010) and run the cleaner. Quick and dirty, but often it works.

As for your request to extract the book meta data from the source code, I have addressed that concern in ticket #5, but did not get around to implementing it yet. (I have a company and am quite busy at the moment, but I will do it one of the next weekends.)

ifix01 commented 11 years ago

well i just tried the workaround you suggested. The book (.zip file) gets unpacked but not cleaned. Instead i see a java exception:

command used on Windows 8 64bit with latest Java VM installed (jar file renamed):

java -jar clean.jar vcsharp_2010

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NullPointerException
        at de.scrum_master.util.FileDownloader.download(FileDownloader.java:107)
        at de.scrum_master.galileo.Downloader.downloadCoverImage(Downloader.java:104)
        at de.scrum_master.galileo.Downloader.download_aroundBody0(Downloader.java:39)
        at de.scrum_master.galileo.Downloader.download_aroundBody1$advice(Downloader.java:30)
        at de.scrum_master.galileo.Downloader.download(Downloader.java:37)
        ...
kriegaex commented 11 years ago

It does not happen here. I saw a similar exception in #9 though and still have no idea why it might happen. Maybe Galileo Press server problems? But this is just a wild guess. Does it also happen with other books? What happens if you switch to single-threaded mode and/or log level 2 or 3?

ifix01 commented 11 years ago

tried again with different log levels (2/3) and single threaded. Always the same exception. With another book again same problem (shell_prog). In that case an empty zip file is created. Seems like there is some problem with downloading:

D:\gal>java -jar clean.jar -l3 shell_prog
    1   Book: shell_prog
    1   >> [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   << [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   >> [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@2ef65160
    1     >> [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@2ef65160
    1     << [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@2ef65160
    1   << [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@2ef65160
    1   >> [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@2ef65160
    1     Downloading, verifying (MD5) and unpacking
    1     >> [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@2ef65160
    1       >> [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       << [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       >> [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@358c908b
    1         >> [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@358c908b
    1         << [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@358c908b
    1       << [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@358c908b
    1       >> [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@358c908b
    1       Downloading http://download2.galileo-press.de/openbook/galileocomputing_shell_programmierung.zip ...
    1       << [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@358c908b
    1     << [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@2ef65160
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@2ef65160
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NullPointerException
        at de.scrum_master.util.FileDownloader.download(FileDownloader.java:107)
        at de.scrum_master.galileo.Downloader.downloadBook(Downloader.java:48)
        at de.scrum_master.galileo.Downloader.download_aroundBody0(Downloader.java:37)
        at de.scrum_master.galileo.Downloader.download_aroundBody1$advice(Downloader.java:30)
        at de.scrum_master.galileo.Downloader.download(Downloader.java:37)
        at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody2(OpenbookCleaner.java:51)
        at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody3$advice(OpenbookCleaner.java:23)
        at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody4(OpenbookCleaner.java:1)
        at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody5$advice(OpenbookCleaner.java:18)
        at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook(OpenbookCleaner.java:1)
        at de.scrum_master.galileo.OpenbookCleaner.main_aroundBody0(OpenbookCleaner.java:29)
        at de.scrum_master.galileo.OpenbookCleaner.main_aroundBody1$advice(OpenbookCleaner.java:12)
        at de.scrum_master.galileo.OpenbookCleaner.main(OpenbookCleaner.java:1)
        ... 5 more
kriegaex commented 11 years ago

How fit are you in Java? Can you debug into the exact place where the exception (NPE) happens? I would, but it is kinda hard to debug into an error which is not reproducable here. I could also add more logging in the problematic section, but if you can help, feel free to do so. GOC is open source... ;-)

ifix01 commented 11 years ago

while i did write some software a few years ago (mainly in C / C++) i've never used Java so far and have no experience with Java debugging, sorry. Still i have some additional information that might help: Today i tried again on my laptop (Win 7 64bit with Java 1.6.0). This also produced the exception however by accident i called the cleaner again and then in worked. So the first call unzipped the file and then crashed. The second call found the already unzipped directory and cleaned it up correctly without any exception. Yesterday in contrast for a "clean start" i always deleted the unzipped directory before trying again with different threading and logging options.

kriegaex commented 11 years ago

What you say is different from what I see in the exception. The first run seems to fail even before the unpacking stage. So please start from scratch by removing the zip and the directory and do the following:

java -jar clean.jar -t0 -l3 shell_prog > log1.txt 2>&1
java -jar clean.jar -t0 -l3 shell_prog > log2.txt 2>&1

This should log both calls including error messages to files log1.txt and log2.txt. Because file uploads are not possible on GitHub, please add two comments, each containing one of the logs as full quotes (copy and paste from a text editor without word wrap). Please do not post logs from your console, they always contain line breaks. And please use code formatting (three backquotes each at the beginning and end of the code section, check the syntax description for GitHub flavored markdown).

ifix01 commented 11 years ago

when i run the two lines for shell_prog (e.g. for a book not already downloaded manually before) i get the exception twice as the download fails. It only works when i use it on an already downloaded book, in my case vcsharp_2010 (which actually is the 2012 edition renamed to 2010 after manually downloading as suggested in your first comment). Here the first call unpacks the zip file and then crashes, while the second call cleans up without exception.

Please let me know if this is enough information. If not i can create the two logfiles for the manually downloaded and renamed book.

kriegaex commented 11 years ago

I created a debug version of 1.0.1. Because I just noticed that GitHub has discontinued its service of uploading binaries, you need to download it from http://scrum-master.de/download/GalileoOpenbookCleaner/galileo_openbook_cleaner-1.0.1-debug.jar. Please run it again with removed zip and unpack directory and send me the log output, so I can see more exactly where the download fails.

ifix01 commented 11 years ago

ok, here's the log file content:

Book: shell_prog
    1   >> [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   << [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   >> [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1     >> [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1     << [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1   << [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1   >> [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@375e9756
  Downloading, verifying (MD5) and unpacking
    1     >> [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@375e9756
    1       >> [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       << [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       >> [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1         >> [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1         << [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1       << [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1       >> [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@292d15a9
    Downloading http://download2.galileo-press.de/openbook/galileocomputing_shell_programmierung.zip ...
    outStream = java.io.FileOutputStream@4ee91f5d
    md5Digest = MD5 Message Digest from SUN, <initialized>

    outStream = [Digest Output Stream] MD5 Message Digest from SUN, <initialized>

    1       << [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@292d15a9
    1     << [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@375e9756
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@375e9756
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NullPointerException
    at de.scrum_master.util.FileDownloader.download(FileDownloader.java:115)
    at de.scrum_master.galileo.Downloader.downloadBook(Downloader.java:48)
    ...
kriegaex commented 11 years ago

Okay, it seems as if something goes wrong in line in = Channels.newChannel(from.openStream()); of class FileDownloader. I have updated the debug code a little bit to get better information. Can you please download the JAR again (same name) and post another debug log? Thank you.

ifix01 commented 11 years ago

no problem; here is the new log:

Book: shell_prog
    1   >> [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   << [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   >> [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1     >> [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1     << [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1   << [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@375e9756
    1   >> [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@375e9756
  Downloading, verifying (MD5) and unpacking
    1     >> [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@375e9756
    1       >> [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       << [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       >> [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1         >> [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1         << [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1       << [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@292d15a9
    1       >> [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@292d15a9
    Downloading http://download2.galileo-press.de/openbook/galileocomputing_shell_programmierung.zip ...
    outStream = java.io.FileOutputStream@4ee91f5d
    md5Digest = MD5 Message Digest from SUN, <initialized>

    outStream = [Digest Output Stream] MD5 Message Digest from SUN, <initialized>

    1       << [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@292d15a9
    1     << [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@375e9756
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@375e9756
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.net.ConnectException: Connection timed out: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.net.URL.openStream(Unknown Source)
    at de.scrum_master.util.FileDownloader.download(FileDownloader.java:90)
    at de.scrum_master.galileo.Downloader.downloadBook(Downloader.java:48)
    at de.scrum_master.galileo.Downloader.download_aroundBody0(Downloader.java:37)
    at de.scrum_master.galileo.Downloader.download_aroundBody1$advice(Downloader.java:30)
    at de.scrum_master.galileo.Downloader.download(Downloader.java:37)
    at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody2(OpenbookCleaner.java:51)
    at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody3$advice(OpenbookCleaner.java:23)
    at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody4(OpenbookCleaner.java:1)
    at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook_aroundBody5$advice(OpenbookCleaner.java:18)
    at de.scrum_master.galileo.OpenbookCleaner.downloadAndCleanBook(OpenbookCleaner.java:1)
    at de.scrum_master.galileo.OpenbookCleaner.main_aroundBody0(OpenbookCleaner.java:29)
    at de.scrum_master.galileo.OpenbookCleaner.main_aroundBody1$advice(OpenbookCleaner.java:12)
    at de.scrum_master.galileo.OpenbookCleaner.main(OpenbookCleaner.java:1)
    ... 5 more
kriegaex commented 11 years ago

Hmm, "connection timed out". Do you have connectivity problems? Or do you need to use a web proxy? That would be an explanation for downloads working in the browser (valid proxy configuration) and not working from the command line.

kriegaex commented 11 years ago

I tried here in office where I also need a proxy. I do not get "connection timed out", but "connection refused", though. Anyway, it might just be a different server configuration. Please try this if you need a proxy (find out the host name or IP and the port first, maybe you even need a user/password, but normally not):

java -Dhttp.proxyHost=[proxy_name_or_ip] -Dhttp.proxyPort=[port_number] -jar galileo_openbook_cleaner-1.0.1-debug.jar unix_guru
ifix01 commented 11 years ago

with my laptop i'm behind a company firewall. Using the proxy with openbook cleaner didn't work (java.io.IOException: Authentication failure). Can try again when i'm at home at my PC, this one has no proxy, it's just behind a Fritz!Box.

kriegaex commented 11 years ago

So we found the problem. Authentication failure means you need a user name and password too. Try to use -Dhttp.proxyUser=[user_name] -Dhttp.proxyPassword=[proxy_password] in addition to the host name and port parameters.

ifix01 commented 11 years ago

still getting the authentication error and i'm sorry i don't have any time left to further investigate into this. What i can do however is try again on my home PC later today.

ifix01 commented 11 years ago

here is the log file from my home PC (direct internet connection via Fritz!Box):

Book: shell_prog
    1   >> [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   << [P] de.scrum_master.galileo.Downloader(File, Book): ---
    1   >> [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@11ec4639
    1     >> [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@11ec4639
    1     << [C] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@11ec4639
    1   << [I] de.scrum_master.galileo.Downloader(File, Book): de.scrum_master.galileo.Downloader@11ec4639
    1   >> [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@11ec4639
  Downloading, verifying (MD5) and unpacking
    1     >> [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@11ec4639
    1       >> [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       << [P] de.scrum_master.util.FileDownloader(URL, File, BigInteger): ---
    1       >> [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@2be2e854
    1         >> [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@2be2e854
    1         << [C] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@2be2e854
    1       << [I] de.scrum_master.util.FileDownloader(URL, File, BigInteger): de.scrum_master.util.FileDownloader@2be2e854
    1       >> [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@2be2e854
    Downloading http://download2.galileo-press.de/openbook/galileocomputing_shell_programmierung.zip ...
    outStream = java.io.FileOutputStream@3e02f94e
    md5Digest = MD5 Message Digest from SUN, <initialized>

    outStream = [Digest Output Stream] MD5 Message Digest from SUN, <initialized>

    1       << [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@2be2e854
    1     << [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@11ec4639
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@11ec4639
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.net.ConnectException: connect: Address is invalid on local machine, or port is not valid on remote machine
    at java.net.DualStackPlainSocketImpl.connect0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.net.URL.openStream(Unknown Source)
    at de.scrum_master.util.FileDownloader.download(FileDownloader.java:90)
    at de.scrum_master.galileo.Downloader.downloadBook(Downloader.java:48)
    ...
ifix01 commented 11 years ago

just one more information: after i disabled my firewall (Kaspersky Internet Security in my case) the download/cleanup was working fine and no exception could be seen in the log file. I guess this is the root cause for all the problems i had on my home PC.

kriegaex commented 11 years ago

Argh! So can I close the ticket? It would be nice if you tried the user/password thing again tomorrow in office, so I can add it to the wiki as an FAQ.

ifix01 commented 11 years ago

ok, you can close it. As for the company laptop i already tried today also with username and password. No idea why it didn't work and i really don't know what else i should try here (and i won't ask our company system administrator for help ...)

kriegaex commented 11 years ago

I just did some web research and found out that the user/password settings are not automatically used by the JVM, in opposite to the host/port settings. I.e. I actually need to change the program code to use those settings. I am going to do that tomorrow or so (just writing this in my hotel on iPad, no PC around here) and upload a new test version. So if you want to give it another five minutes in office (without abusing your sysadmin), I will be glad. I cannot test it myself, having no authenticating proxy around here. Otherwise I would need to set up one by myself just for testing's sake.

ifix01 commented 11 years ago

no problem, i can run another test. Please tell me when the updated SW is ready. And one question: Is there any special syntax for passwords containing special characters ?

kriegaex commented 11 years ago

I have added support for non-authenticating as well as authenticating (with user/password) HTTP web proxies: http://scrum-master.de/download/GalileoOpenbookCleaner/galileo_openbook_cleaner-1.0.2.jar

Please give it a try and tell me if it works now. The command line could be as follows (no line breaks!):

java
    -Dhttp.proxyHost=localhost
    -Dhttp.proxyPort=8080
    -Dhttp.proxyUser=kriegaex
    -Dhttp.proxyPassword="test äöü"
    -jar galileo_openbook_cleaner-1.0.2.jar
    unix_guru

Please replace with your respective proxy settings for host, port, user, password.

ifix01 commented 11 years ago

still no download and an exception:

    ...
    Unzipping archive .\galileocomputing_unix_guru.zip ...
    1       << [M] void de.scrum_master.util.ZipFileExtractor.unzip(): de.scrum_master.util.ZipFileExtractor@2cb0ce8f
    1     << [M] void de.scrum_master.galileo.Downloader.unpackBook(): de.scrum_master.galileo.Downloader@77827284
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@77827284
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.NullPointerException
    at de.scrum_master.util.ZipFileExtractor.unzip(ZipFileExtractor.java:63)
    at de.scrum_master.galileo.Downloader.unpackBook(Downloader.java:64)
    ...
ifix01 commented 11 years ago

sorry didn't delete the empty zip file before starting. Here is a "clean run" but it shows another authentication problem:

    ...
    1       >> [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@624b035d
    Downloading http://download2.galileo-press.de/openbook/galileocomputing_unix_guru.zip ...
    1       << [M] void de.scrum_master.util.FileDownloader.download(): de.scrum_master.util.FileDownloader@624b035d
    1     << [M] void de.scrum_master.galileo.Downloader.downloadBook(): de.scrum_master.galileo.Downloader@375e9756
    1   << [M] void de.scrum_master.galileo.Downloader.download(): de.scrum_master.galileo.Downloader@375e9756
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.io.IOException: Authentication failure
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at de.scrum_master.util.FileDownloader.download(FileDownloader.java:125)
    at de.scrum_master.galileo.Downloader.downloadBook(Downloader.java:48)
    ...
kriegaex commented 11 years ago

Looks like your proxy user/password is somehow wrong. What is your command line? If your password contains special characters like space, did you enclose it in double quotes as shown in my example?

ifix01 commented 11 years ago

yes, password is enclosed in double quotes. Not sure however if all the data is correct as normally i don't have to use this stuff at all (it's auto configured in the Internet Explorer via config script and again i can't discuss with our sysadmin).

kriegaex commented 11 years ago

Too bad you cannot talk to him. Is he so scary? You are permitted to download data via web browser, why should it be a problem to download something from a Java application? Anyway, if you cannot talk to him and ask what kind of authentication scheme is used and if it is authenticating (and which auth type) or not, the only thing we could do is guess, try several options and find the solution by chance. I would like to further improve the program and help you use it in office, but probably like this we have reached a dead end, unless you can talk to him. If you do, feel free to re-open the ticket, I would be glad to assist.

Edit: I have tested authentication at home with an extra installed proxy. It worked nicely.

kriegaex commented 11 years ago

Info: In the developer version (branch master, commit 3593c814 or newer) the book list is now configurable, just in case you want to test it. The next version (not released yet) will include this feature, a preview version is available at http://scrum-master.de/download/GalileoOpenbookCleaner/galileo_openbook_cleaner-dev.jar.

kriegaex commented 11 years ago

Contained in the new release 1.1.0, see main page.