Downloading Schedule directly from iMaalum using student credentials

[PlashSpeed-Aiman]

I've been trying to figure out how to download the confirmation slip directly from iMaalum's website using POST requests. It seems I need to find out how to do GET request with cookies.

[iqfareez]

Hmm. No idea, never did that before. I believe it is trickier due to the CAS authentication.

[PlashSpeed-Aiman]

I did it using Go to login to the website and get the login cookies. However, when trying to get a specific file, it returns the main webpage.

So, in order to succeed in login, send a GET request to the server to get the first cookie, and send a POST request with credentials to get the second cookie and successfully login to iMaalum. The remaining problem is requesting the schedule file

resp_first,_ := client.Get("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome")
    client.Jar.SetCookies(urlObj,resp_first.Cookies())
    resp,_ := client.PostForm("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome",formVal)
    // client.Jar.SetCookies(urlObj,resp.Cookies())
    resp_get,_:=client.Get("https://imaluum.iium.edu.my/MyAcademic/resultprint?")

[PlashSpeed-Aiman]

It's working now. You just have to translate the code to Dart and after that you can get the files directly from iMaalum

check it out at IIUMPassGo

urlObj,_ := url.Parse("https://imaluum.iium.edu.my/")
resp_first,_ := client.Get("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome")
client.Jar.SetCookies(urlObj,resp_first.Cookies())
cookies1 := resp_first.Cookies()
resp,_ := client.PostForm("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome",formVal)
newCook:=append(cookies1, resp.Cookies()...)
client.Jar.SetCookies(urlObj,newCook)
resp_get,_ :=client.Get("https://imaluum.iium.edu.my/confirmationslip?ses=2022/2023&sem=1")

[iqfareez]

Mah mann, I tried them just now and it just works! Porting this into Dart...

[iqfareez]

Hey @PlashSpeed-Aiman. I'm working on porting the Dart code but stuck a bit, wondering if you could shed some light ahha

Go

In the Go code, I tried to inspect the outgoing request to understand the cookies etc. So for first GET request.

    resp_first, _ := client.Get("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome")

    fmt.Println("resp_first Header")
    fmt.Println(resp_first.Header)
    fmt.Println()

Output:

map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Language:[en] Content-Type:[text/html;charset=UTF-8] Date:[Sun, 25 Dec 2022 05:06:16 GMT] Expires:[0] Pragma:[no-cache] Server:[Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_nss/1.0.14 NSS/3.28.4 PHP/5.4.16] Set-Cookie:[SESSION=19057333-0ec9-4dc4-a08f-d030e41cab53;path=/cas/;Secure;HttpOnly YpN9wASukHaMGkYA=v1cyFLgw__wKC; Expires=Sun, 25-Dec-2022 07:06:16 GMT; Path=/; Secure] Strict-Transport-Security:[max-age=15768000 ; includeSubDomains] Vary:[Accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1; mode=block]]

For second request (POST)

    client.Jar.SetCookies(urlObj, resp_first.Cookies())
    cookies1 := resp_first.Cookies()
    resp, _ := client.PostForm("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome", formVal)
    fmt.Println("resp Request Header")
    fmt.Println(resp.Request.Header)

Output:

resp Request Header
map[Content-Type:[application/x-www-form-urlencoded] Cookie:[YpN9wASukHaMGkYA=v1cyFLgw__wKC; MOD_AUTH_CAS=344c7f76cf41a9eefac3ef03607da9d0] Referer:[https://imaluum.iium.edu.my/home?ticket=ST-5718104-9NtiIQxqQ4wVngG6WSd--OGJoxo-cas2]]

So, I wonder here does MOD_AUTH_CAS come from in the second request? It doesn't appear in the first cookie response.

Dart

In Dart, I got 500 error while POST-ing the second request

```dart void login(String username, String password) async { var dio = Dio(); var cookieJar = CookieJar(); await cookieJar.deleteAll(); dio.interceptors.add(CookieManager(cookieJar)); var imaluumUri = Uri.parse('https://imaluum.iium.edu.my/'); var casUrl = 'https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome'; //Get first cookies var res1 = await dio.get(casUrl); var parsedCookies = res1.headers['set-cookie']?.map((e) { var cookie = Cookie.fromSetCookieValue(e); return cookie; }).toList(); print('First response cookies:'); print(parsedCookies); // save cookies await cookieJar.saveFromResponse(imaluumUri, parsedCookies!); // print(await cookieJar.loadForRequest(imaluumUri)); var formVal = { 'username': username, 'password': password, 'execution': 'e1s1', '_eventId': 'submit', 'geolocation': '', }; FormData formData = FormData.fromMap(formVal); // print(formData.fields); Response? resAuth; // second request with the cookie try { resAuth = await dio.post( 'https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome', data: formData, options: Options(contentType: Headers.formUrlEncodedContentType), ); } on DioError catch (e) { print('Second request'); print(e.requestOptions.headers); } print(resAuth); // print(await cookieJar.loadForRequest(imaluumUri)); return; } ```

Output:

First response cookies:
[SESSION=30f87fbc-76de-462b-942b-c0948a09c7ff; Path=/cas/; Secure; HttpOnly, YpN9wASukHaMGkYA=v1cyFLgw__wKC; Expires=Sun, 25 Dec 2022 07:14:38 GMT; Path=/; Secure]
Second request
{content-type: multipart/form-data; boundary=--dio-boundary-3844012906, cookie: SESSION=30f87fbc-76de-462b-942b-c0948a09c7ff; YpN9wASukHaMGkYA=v1cyFLgw__wKC, content-length: 550}

I believe because it doesn't send along with the MOD_AUTH_CAS cookie

[PlashSpeed-Aiman]

Try changing await cookieJar.saveFromResponse(imaluumUri, parsedCookies!); to await cookieJar.saveFromResponse(casUrl, parsedCookies!); and save all cookies from POST request because in the Go implementation I saved both cookies from GET and POST. Those cookies will be used when downloading files etc

[PlashSpeed-Aiman]

I think it's better to use http or dart http libraries compared to DIO. I read in some places that DIO doesn't work most of the time.

[iqfareez]

I see. Yeah, that might be the case. Thank you.

[ElyasAsmad]

Hey @PlashSpeed-Aiman. I'm working on porting the Dart code but stuck a bit, wondering if you could shed some light ahha

Go

In the Go code, I tried to inspect the outgoing request to understand the cookies etc. So for first GET request.

  resp_first, _ := client.Get("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome")

  fmt.Println("resp_first Header")
  fmt.Println(resp_first.Header)
  fmt.Println()

Output:

map[Cache-Control:[no-cache, no-store, max-age=0, must-revalidate] Content-Language:[en] Content-Type:[text/html;charset=UTF-8] Date:[Sun, 25 Dec 2022 05:06:16 GMT] Expires:[0] Pragma:[no-cache] Server:[Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_nss/1.0.14 NSS/3.28.4 PHP/5.4.16] Set-Cookie:[SESSION=19057333-0ec9-4dc4-a08f-d030e41cab53;path=/cas/;Secure;HttpOnly YpN9wASukHaMGkYA=v1cyFLgw__wKC; Expires=Sun, 25-Dec-2022 07:06:16 GMT; Path=/; Secure] Strict-Transport-Security:[max-age=15768000 ; includeSubDomains] Vary:[Accept-encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1; mode=block]]

For second request (POST)

  client.Jar.SetCookies(urlObj, resp_first.Cookies())
  cookies1 := resp_first.Cookies()
  resp, _ := client.PostForm("https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome", formVal)
  fmt.Println("resp Request Header")
  fmt.Println(resp.Request.Header)

Output:

resp Request Header
map[Content-Type:[application/x-www-form-urlencoded] Cookie:[YpN9wASukHaMGkYA=v1cyFLgw__wKC; MOD_AUTH_CAS=344c7f76cf41a9eefac3ef03607da9d0] Referer:[https://imaluum.iium.edu.my/home?ticket=ST-5718104-9NtiIQxqQ4wVngG6WSd--OGJoxo-cas2]]

So, I wonder here does MOD_AUTH_CAS come from in the second request? It doesn't appear in the first cookie response.

Dart

In Dart, I got 500 error while POST-ing the second request

```dart void login(String username, String password) async { var dio = Dio(); var cookieJar = CookieJar(); await cookieJar.deleteAll(); dio.interceptors.add(CookieManager(cookieJar)); var imaluumUri = Uri.parse('https://imaluum.iium.edu.my/'); var casUrl = 'https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome'; //Get first cookies var res1 = await dio.get(casUrl); var parsedCookies = res1.headers['set-cookie']?.map((e) { var cookie = Cookie.fromSetCookieValue(e); return cookie; }).toList(); print('First response cookies:'); print(parsedCookies); // save cookies await cookieJar.saveFromResponse(imaluumUri, parsedCookies!); // print(await cookieJar.loadForRequest(imaluumUri)); var formVal = { 'username': username, 'password': password, 'execution': 'e1s1', '_eventId': 'submit', 'geolocation': '', }; FormData formData = FormData.fromMap(formVal); // print(formData.fields); Response? resAuth; // second request with the cookie try { resAuth = await dio.post( 'https://cas.iium.edu.my:8448/cas/login?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome?service=https%3a%2f%2fimaluum.iium.edu.my%2fhome', data: formData, options: Options(contentType: Headers.formUrlEncodedContentType), ); } on DioError catch (e) { print('Second request'); print(e.requestOptions.headers); } print(resAuth); // print(await cookieJar.loadForRequest(imaluumUri)); return; } ```

Output:

First response cookies:
[SESSION=30f87fbc-76de-462b-942b-c0948a09c7ff; Path=/cas/; Secure; HttpOnly, YpN9wASukHaMGkYA=v1cyFLgw__wKC; Expires=Sun, 25 Dec 2022 07:14:38 GMT; Path=/; Secure]
Second request
{content-type: multipart/form-data; boundary=--dio-boundary-3844012906, cookie: SESSION=30f87fbc-76de-462b-942b-c0948a09c7ff; YpN9wASukHaMGkYA=v1cyFLgw__wKC, content-length: 550}

I believe because it doesn't send along with the MOD_AUTH_CAS cookie

I finally managed to solve the MOD_AUTH_CAS problem. The following PR will be submitted later.

Apparently, the CAS server will return the MOD_AUTH_CAS cookie inside the 302 HTTP response after executing a GET request on the url returned by the server after a successful login attempt but somehow, Dart's http library skipped that cookie. Consequently, i-Maluum redirects any subsequent requests to the CAS page. By setting HttpClientRequest.followRedirects = false, we can easily get the MOD_AUTH_CAS cookie.

[ElyasAsmad]

However, I believe the current implementation of parsing schedules directly from i-Maluum using WebView is much more friendly than requiring users to enter their matric number and password into our text fields. Users will get a sense of safety and trust as they used to see i-Maluum interface and login into that platform and maybe not trusting any other platform.

A simple A/B testing can be done to justify my theory

[iqfareez]

Apparently, the CAS server will return the MOD_AUTH_CAS cookie inside the 302 HTTP response after executing a GET request on the url returned by the server after a successful login attempt but somehow, Dart's http library skipped that cookie. Consequently, i-Maluum redirects any subsequent requests to the CAS page. By setting HttpClientRequest.followRedirects = false, we can easily get the MOD_AUTH_CAS cookie.

Ahh I see. Thank you Elyas.

I believe the current implementation of parsing schedules directly from i-Maluum using WebView is much more friendly than requiring users to enter their matric number and password into our text fields.

Agreed 💯 . We'll keep it as default. Perhaps, the method we've discussed here can be useful on web/windows because WebView is not supported there.