search

ucsb-cs56-projects / cs56-scrapers-ucsb-curriculum

-
1 stars 12 forks source link

URL has moved to https, so must change code to use HTTPS. #4

Closed pconrad closed 10 years ago

pconrad commented 11 years ago

Turning on debug, we see that the URL being retrieved by the Java code is returning the page:

run:
     [java] extractHiddenFieldValue: page=<html><head><title>Moved Permanently</title></head><body><p>The URI that you requested has been <a href="https://my.sa.ucsb.edu/public/curriculum/coursesearch.aspx">moved to here</a>.</p></body></html>

Here are a few links from Google searching "java https" that might be helpful.

http://stackoverflow.com/questions/6927427/how-to-send-https-post-request-in-java

http://www.mkyong.com/java/java-https-client-httpsurlconnection-example/

pconrad commented 11 years ago

This might also help: http://stackoverflow.com/questions/13626965/how-to-ignore-pkix-path-building-failed-sun-security-provider-certpath-suncertp

pconrad commented 11 years ago

See also: http://www.cs.ucsb.edu/~pconrad/cs56/examples/ldap/SimpleQuery/InstallCert.java for a program that installs certificates.

Phillip-Conrads-Mac-mini:cs56-scrapers-ucsb-curriculum pconrad$ java -cp build InstallCert my.sa.ucsb.edu:443
Loading KeyStore /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/security/cacerts...
Opening connection to my.sa.ucsb.edu:443...
Starting SSL handshake...

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1747)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
    at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1209)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:135)
    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
    at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:943)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
    at InstallCert.main(InstallCert.java:91)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:323)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:217)
    at sun.security.validator.Validator.validate(Validator.java:218)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
    at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
    at InstallCert$SavingTrustManager.checkServerTrusted(InstallCert.java:186)
    at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1201)
    ... 8 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:318)
    ... 14 more

Server sent 1 certificate(s):

 1 Subject CN=my.sa.ucsb.edu, OU=Student Affairs, O=University of California Santa Barbara, L=Santa Barbara, ST=California, C=US, SERIALNUMBER=ze2vYQkwbffm7ek-fnqU9nu/0cGx6-72
   Issuer  CN=GeoTrust SSL CA, O="GeoTrust, Inc.", C=US
   sha1    bb 91 d5 1f 7a 2f 61 9b 71 97 ef a5 a8 bb 58 e5 77 06 dd bb 
   md5     8b 05 fc 02 2d 4a 88 6f d5 9a 0e bb 0e e0 bd 61 

Enter certificate to add to trusted keystore or 'q' to quit: [1]

[
[
  Version: V3
  Subject: CN=my.sa.ucsb.edu, OU=Student Affairs, O=University of California Santa Barbara, L=Santa Barbara, ST=California, C=US, SERIALNUMBER=ze2vYQkwbffm7ek-fnqU9nu/0cGx6-72
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 29622016342401222549139234264854358669069774701855182051237196024238578495766092502312945878888632559242524483749747828130883046558445951209199969796804627135753186691697037160179750767260744383846787641491686564250294579460767384320653486191273067356209118380557772475326362973999240632746345636813871294481770906471740260352038904622525961473759463506455318210840949505683226663746632876682751457593653615538949126617276999208009999763604031307704148257077294974505952580026331045775396555918188363477914967452244919463463958543524029327783727315991838377529472695484303291205511625541655877995746920664528793847903
  public exponent: 65537
  Validity: [From: Sun Sep 11 23:38:08 PDT 2011,
               To: Sat Dec 13 16:47:37 PST 2014]
  Issuer: CN=GeoTrust SSL CA, O="GeoTrust, Inc.", C=US
  SerialNumber: [    dd6c]

Certificate Extensions: 8
[1]: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: 1.3.6.1.5.5.7.48.2
   accessLocation: URIName: http://gtssl-aia.geotrust.com/gtssl.crt]
]

[2]: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 42 79 54 1B 61 CD 55 2B   3E 63 D5 3C 48 57 F5 9F  ByT.a.U+>c.<HW..
0010: FB 45 CE 4A                                        .E.J
]

]

[3]: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:false
  PathLen: undefined
]

[4]: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://gtssl-crl.geotrust.com/crls/gtssl.crl]
]]

[5]: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

[6]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
  Data_Encipherment
]

[7]: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: my.sa.ucsb.edu
]

[8]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 55 9D E5 0A 5D E7 9A ED   2A 19 A0 8E 44 70 A0 25  U...]...*...Dp.%
0010: FF 38 0A 4A                                        .8.J
]
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 82 BA E5 DF FC 32 79 A7   3B 05 24 71 41 16 5B 33  .....2y.;.$qA.[3
0010: 3F E1 2D 3C 05 40 43 48   E2 D0 86 1A 44 44 10 CB  ?.-<.@CH....DD..
0020: F8 0F C6 8D 04 64 D6 EC   4A A9 C8 C6 A0 43 99 FA  .....d..J....C..
0030: 05 F8 8D 03 A4 83 A7 04   53 50 F8 D5 D2 4A 0D AE  ........SP...J..
0040: 84 A1 00 04 56 A7 41 86   04 34 65 7F 76 E0 60 6D  ....V.A..4e.v.`m
0050: 98 38 39 27 9E D4 65 75   7B 4F D0 57 A1 E7 05 E4  .89'..eu.O.W....
0060: 31 62 D7 CE CE A3 31 69   DE 3F E2 30 D2 B5 02 FF  1b....1i.?.0....
0070: 8F 7C 96 4D 84 FC D4 BF   E7 13 D4 CE 74 C1 B5 16  ...M........t...
0080: 63 DA 61 19 C1 28 AA E2   6F 16 66 F1 5D 32 D0 6F  c.a..(..o.f.]2.o
0090: D7 CE 1D 43 3E BC 0A AB   6B 52 D6 A4 0D CE A0 5A  ...C>...kR.....Z
00A0: CA 05 8C 81 7F 29 BC CB   28 24 13 E3 2E FC A7 90  .....)..($......
00B0: A6 FD 59 A6 B1 D7 8A 50   3D 98 41 9F 44 9D F6 CE  ..Y....P=.A.D...
00C0: 3C C8 83 1C 9E 6E 82 92   2B E4 61 80 EE C4 D4 AA  <....n..+.a.....
00D0: BB 8A 1F FF 2E 25 C5 6A   F6 69 C1 CC 62 BC 98 3D  .....%.j.i..b..=
00E0: 1F 03 88 F2 CD 82 A1 C4   1A 48 CF 3E 34 17 4B 20  .........H.>4.K 
00F0: 53 A6 0B 5A D1 2B 16 D7   25 47 21 A9 8A 8C BC 10  S..Z.+..%G!.....

]

Added certificate to keystore 'jssecacerts' using alias 'my.sa.ucsb.edu-1'
Phillip-Conrads-Mac-mini:cs56-scrapers-ucsb-curriculum pconrad$
nguyenmp commented 11 years ago

I would like to claim this issue. @CommanderHamster, please assign it to me!

CommanderHamster commented 11 years ago

~estimated 200

pconrad commented 11 years ago

The HTTPS part is done, but there is still 200 points of work to do because it still has some new bug.

CARammelkamp commented 11 years ago

~claimed @nguyenmp ~moderator @CommanderHamster

nguyenmp commented 11 years ago

@dvicory and I ended up modifying our build.xml to do the ssl certificate download. Here is the commit.

This commit changed the build.xml file so it automatically accepts the certificate without prompt.

It will be included in our pull request when we are all done.

pconrad commented 11 years ago

Terrific! On May 14, 2013 12:10 PM, "Mark Nguyen" notifications@github.com wrote:

@dvicory https://github.com/dvicory and I ended up modifying our build.xml to do the ssl certificate download. Herehttps://github.com/nguyenmp/cs56-scrapers-ucsb-curriculum/commit/b0034bd693c3cee53c8931985622ce2786fb28e2is the commit.

Thishttps://github.com/nguyenmp/cs56-scrapers-ucsb-curriculum/commit/ddc80cc07ce75e54141f6745802fe535d1b6b11dcommit changed the build.xml file so it automatically accepts the certificate without prompt.

It will be included in our pull request when we are all done.

— Reply to this email directly or view it on GitHubhttps://github.com/UCSB-CS56-Projects/cs56-scrapers-ucsb-curriculum/issues/4#issuecomment-17898176 .