How to perform an "APPEND" to a remote file

[ricbrue]

Hi, I would like to use this library for a project where I need to send "APPEND" commands to a remote file, does this library allows this operation?

[embeddedmz]

Hi,

I don't know this command but I found this : https://stackoverflow.com/questions/26662538/curl-add-content-to-an-existing-file-using-one-request and this https://curl.se/docs/manpage.html#-a

From this : https://curl.se/libcurl/c/CURLOPT_APPEND.html and from what I understood of your need, I wrote this new upload method (AppendFile) that takes the offset of the file you want to upload from :

bool CFTPClient::AppendFile(const std::string &strLocalFile, const size_t fileOffset, const std::string &strRemoteFile, const bool &bCreateDir) const {
   if (strLocalFile.empty() || strRemoteFile.empty()) return false;

   if (!m_pCurlSession) {
      if (m_eSettingsFlags & ENABLE_LOG) m_oLog(LOG_ERROR_CURL_NOT_INIT_MSG);

      return false;
   }
   // Reset is mandatory to avoid bad surprises
   curl_easy_reset(m_pCurlSession);

   std::ifstream InputFile;
   std::string strLocalRemoteFile = ParseURL(strRemoteFile);

   struct stat file_info;
   bool bRes = false;

   /* get the file size of the local file */
   #ifdef LINUX
   if (stat(strLocalFile.c_str(), &file_info) == 0) {
      InputFile.open(strLocalFile, std::ifstream::in | std::ifstream::binary);
   #else
   static_assert(sizeof(struct stat) == sizeof(struct _stat64i32), "Oh oh !");
   std::wstring wstrLocalFile = Utf8ToUtf16(strLocalFile);
   if (_wstat64i32(wstrLocalFile.c_str(), reinterpret_cast<struct _stat64i32*>(&file_info)) == 0) {
      InputFile.open(wstrLocalFile, std::ifstream::in | std::ifstream::binary);
   #endif
      if (!InputFile) {
         if (m_eSettingsFlags & ENABLE_LOG) m_oLog(StringFormat(LOG_ERROR_FILE_UPLOAD_FORMAT, strLocalFile.c_str()));

         return false;
      }

      // check of the offset is less than the file size
      if (fileOffset >= file_info.st_size)
      {
          if (m_eSettingsFlags & ENABLE_LOG) m_oLog("ERROR Incorrect offset !"); // if this code is OK use existing coding style for log msgs
          return false;
      }

      InputFile.seekg(fileOffset, InputFile.beg); // Sets the position of the next character to be extracted from the input stream.

      /* specify target */
      curl_easy_setopt(m_pCurlSession, CURLOPT_URL, strLocalRemoteFile.c_str());

      /* we want to use our own read function */
      curl_easy_setopt(m_pCurlSession, CURLOPT_READFUNCTION, ReadFromFileCallback);

      /* now specify which file to upload */
      curl_easy_setopt(m_pCurlSession, CURLOPT_READDATA, &InputFile);

      /* Set the size of the file to upload (optional).  If you give a *_LARGE
      option you MUST make sure that the type of the passed-in argument is a
      curl_off_t. If you use CURLOPT_INFILESIZE (without _LARGE) you must
      make sure that to pass in a type 'long' argument. */
      curl_easy_setopt(m_pCurlSession, CURLOPT_INFILESIZE_LARGE, static_cast<curl_off_t>(file_info.st_size - fileOffset)); // Important !

      /* enable uploading */
      curl_easy_setopt(m_pCurlSession, CURLOPT_UPLOAD, 1L);
      curl_easy_setopt(m_pCurlSession, CURLOPT_APPEND, 1L);

      if (bCreateDir) curl_easy_setopt(m_pCurlSession, CURLOPT_FTP_CREATE_MISSING_DIRS, CURLFTP_CREATE_DIR);

      // TODO add the possibility to rename the file upon upload finish....

      CURLcode res = Perform();

      if (res != CURLE_OK) {
         if (m_eSettingsFlags & ENABLE_LOG)
            m_oLog(StringFormat(LOG_ERROR_CURL_UPLOAD_FORMAT, strLocalFile.c_str(), res, curl_easy_strerror(res)));
      } else
         bRes = true;
   }
   InputFile.close();

   return bRes;
}

Can you test it and give me a feedback ? Compare the hash of the uploaded file with the original one to check if the method succeeded in uploading the file (e.g. split a file in two using a C/C++/C# program, upload the two parts in the right order to a remote file the first part with the classic UploadFile method whereas for the second part use this new method with an offset equal to the half of the size of the file and must be consistent with the program you used to split the test file, compare the SHA1 of the uploaded file with the one that you split).

I will try to run a unit test with a local FTP server (Cerberus FTP Server). I don't know if this command is supported !

Thanks.

[embeddedmz]

I have made a quick and dirty unit test, I have tested with Cerberus FTP Server (for the FTP protocol) and Rebex Tiny SFTP Server and the code above works fine with both protocols (FTP and SFTP).

[embeddedmz]

The code is now available in the main branch.