update libcurl with SSL support

[bucanero]

This PR updates libcurl to 7.64.1, and uses polarSSL for TLS 1.2 support

[sharkwouter]

Oh cool! That's a great project. I had no idea this could work with such a small patch. I used an old port and made the patch by doing a diff. I have no idea what was even in it.

Do you know if this updated curl library works better than the old one? Do you have some code to test it with?

[bucanero]

I ported this curl 7.64.1 +polarSSL version to PS4 (open orbis sdk) and it works nice, so I thought about updating the PSP build too. I did a quick overlook on the 7.15.1 patch, and most changes seem not relevant anymore, so I created a minimal patch to use cmake with polarSSL

I still need to test this on real PSP hardware, just to be sure everything is still working as expected. If nothing weird happens, this should be a nice feature bump from 7.15.1

About test code, this small function should be enough to test, once the PSP app has a network link:

http_download("https://google.com/robots.txt", "ms0:/robots.txt");

#define HTTP_SUCCESS 1
#define HTTP_FAILED  0

int http_download(const char* full_url, const char* local_dst)
{
    CURL *curl;
    CURLcode res;
    FILE* fd;

    curl = curl_easy_init();
    if(!curl)
    {
        LOG("ERROR: CURL INIT");
        return HTTP_FAILED;
    }

    fd = fopen(local_dst, "wb");
    if (!fd) {
        printf("fopen Error: File path '%s'", local_dst);
        return HTTP_FAILED;
    }

    printf("Download URL: %s >> %s", full_url, local_dst);

    curl_easy_setopt(curl, CURLOPT_URL, full_url);
    // Set user agent string
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "cURL agent");
    // not sure how to use this when enabled
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
    // not sure how to use this when enabled
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
    // Set SSL VERSION to TLS 1.2
    curl_easy_setopt(curl, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2);
    // Set timeout for the connection to build
    curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 10L);
    // Follow redirects (?)
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
    // The function that will be used to write the data 
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
    // The data filedescriptor which will be written to
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, fd);
    // maximum number of redirects allowed
    curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 20L);
    // Fail the request if the HTTP code returned is equal to or larger than 400
    curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
    // request using SSL for the FTP transfer if available
    curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_TRY);

    // Perform the request
    res = curl_easy_perform(curl);
    // close filedescriptor
    fclose(fd);
    // cleanup
    curl_easy_cleanup(curl);

    if(res != CURLE_OK)
    {
        printf("curl_easy_perform() failed: %s", curl_easy_strerror(res));
        remove(local_dst);
        return HTTP_FAILED;
    }

    return HTTP_SUCCESS;
}

Edit: if missing basename() function when linking, add:

char * basename (const char *filename)
{
    char *p = strrchr (filename, '/');
    return p ? p + 1 : (char *) filename;
}

[bucanero]

hi @sharkwouter , quick question, what's the recommended SDK package to download from the pspdev releases? should I use v20200725-2 (tagged as latest) ? or maybe something newer, like Development build from Nov16, 2021 "pre-release"

I'm building a simple app to test this updated libcurl with SSL support on real hardware, and I want to be sure I'm using the best stable sdk, to closely replicate the Docker environment you're using here on the auto-build.

[sharkwouter]

I'd recommend to always get the development build. We don't really make new tags at the moment.

[KorigamiK]

So I am testing this PR on this branch (I know it's not a very minimal example) the only new libraries I had to link against were:

z
polarssl
curl

However compiling fails due to the following error:

build] [ 16%] Linking CXX executable curl_test
[build] /usr/local/pspdev/lib/gcc/psp/11.2.0/../../../../psp/bin/ld: /usr/local/pspdev/psp/lib/libcurl.a(mime.c.obj): in function `curl_mime_filedata':
[build] (.text+0x1c9c): undefined reference to `basename'
[build] collect2: error: ld returned 1 exit status
[build] make[2]: *** [CMakeFiles/curl_test.dir/build.make:161: curl_test] Error 1
[build] make[1]: *** [CMakeFiles/Makefile2:84: CMakeFiles/curl_test.dir/all] Error 2

A quick google search lead to this answer so I wonder if something similar is required here.

[KorigamiK]

I ported this curl 7.64.1 +polarSSL version to PS4 (open orbis sdk) and it works nice, so I thought about updating the PSP build too. I did a quick overlook on the 7.15.1 patch, and most changes seem not relevant anymore, so I created a minimal patch to use cmake with polarSSL

I still need to test this on real PSP hardware, just to be sure everything is still working as expected. If nothing weird happens, this should be a nice feature bump from 7.15.1

About test code, this small function should be enough to test, once the PSP app has a network link:

http_download("https://google.com/robots.txt", "ms0:/robots.txt");

#define HTTP_SUCCESS 1
#define HTTP_FAILED  0

int http_download(const char* full_url, const char* local_dst)
{
  CURL *curl;
  CURLcode res;
  FILE* fd;

  curl = curl_easy_init();
  if(!curl)
  {
      LOG("ERROR: CURL INIT");
      return HTTP_FAILED;
  }

  fd = fopen(local_dst, "wb");
  if (!fd) {
      printf("fopen Error: File path '%s'", local_dst);
      return HTTP_FAILED;
  }

  printf("Download URL: %s >> %s", full_url, local_dst);

  curl_easy_setopt(curl, CURLOPT_URL, full_url);
  // Set user agent string
  curl_easy_setopt(curl, CURLOPT_USERAGENT, "cURL agent");
  // not sure how to use this when enabled
  curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
  // not sure how to use this when enabled
  curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
  // Set SSL VERSION to TLS 1.2
  curl_easy_setopt(curl, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_2);
  // Set timeout for the connection to build
  curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 10L);
  // Follow redirects (?)
  curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
  // The function that will be used to write the data 
  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
  // The data filedescriptor which will be written to
  curl_easy_setopt(curl, CURLOPT_WRITEDATA, fd);
  // maximum number of redirects allowed
  curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 20L);
  // Fail the request if the HTTP code returned is equal to or larger than 400
  curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
  // request using SSL for the FTP transfer if available
  curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_TRY);

  // Perform the request
  res = curl_easy_perform(curl);
  // close filedescriptor
  fclose(fd);
  // cleanup
  curl_easy_cleanup(curl);

  if(res != CURLE_OK)
  {
      printf("curl_easy_perform() failed: %s", curl_easy_strerror(res));
      remove(local_dst);
      return HTTP_FAILED;
  }

  return HTTP_SUCCESS;
}

I tried to use this exact function here.

[bucanero]

So I am testing this PR on this branch (I know it's not a very minimal example) the only new libraries I had to link against were:

z
polarssl
curl

However compiling fails due to the following error:

build] [ 16%] Linking CXX executable curl_test
[build] /usr/local/pspdev/lib/gcc/psp/11.2.0/../../../../psp/bin/ld: /usr/local/pspdev/psp/lib/libcurl.a(mime.c.obj): in function `curl_mime_filedata':
[build] (.text+0x1c9c): undefined reference to `basename'
[build] collect2: error: ld returned 1 exit status
[build] make[2]: *** [CMakeFiles/curl_test.dir/build.make:161: curl_test] Error 1
[build] make[1]: *** [CMakeFiles/Makefile2:84: CMakeFiles/curl_test.dir/all] Error 2

A quick google search lead to this answer so I wonder if something similar is required here.

thanks for your tests @KorigamiK

Yes, basename() is missing from the psp-sdk, but is still defined in the standard headers. It's a very simple function, so it can be ported to the PSP with no issues.

Until it's added to the psp-sdk, you can add the code to network.cpp:

extern "C" {
char * basename (const char *filename)
{
    char *p = strrchr (filename, '/');
    return p ? p + 1 : (char *) filename;
}

}

That should be enough to link and compile everything (curl, polarssl, z)