Error Loading CSV Data Using Bulk Data Loader 2.0

[stevenwoodson]

I realize that this library may not necessarily have been intended for use with the Bulk API 2.0 but I've gotten so very close and enjoy using this for everything else.

I've been following these very detailed instructions on using the Bulk API and have realized that the issue lies in the step where the CSV data is pushed using PUT. Digging into what Salesforce receives, it looked like the raw CSV data is json_encoded before sending which renders it unusable. I've isolated the issue to this line that seems to assume that all body data in every request needs to be json_encoded before sending.

As a proof of concept, I added a conditional around that formatter->setBody step to check that the Content-Type is application/json before formatting and sure enough that cleared the issue for me because my data should be sent with Type text/csv. Here's the relevant block of code I had changed:

        if (isset($this->options['body'])) {
            if ( $this->parameters['headers']['Content-Type'] == 'application/json') {
                $this->parameters['body'] = $this->formatter->setBody($this->options['body']);
            } else {
                $this->parameters['body'] = $this->options['body'];
            }
        } else {
            unset($this->parameters['body']);
        }

Obviously, this isn't the most reliable or intelligent way to do this but I fear I don't know this codebase quite well enough to know for sure how this should be handled, which is why I opened an issue instead of a PR.

---

For more context, here's the relevant code I'm using to follow the Create/Add Data/Close steps noted in the instructions I linked to above.

Create

        $response = Forrest::jobs('ingest', [
            'method' => 'post',
            'body' => [
                "object" => "Contact",
                "externalIdFieldName" => "Id",
                "contentType" => "CSV",
                "operation" => "upsert"
            ]
        ]);

Add Data

        $response = Forrest::jobs('ingest/' . $job['id'] . '/batches', [
            'method' => 'put',
            'headers' => [
                'Content-Type' => 'text/csv'
            ],
            'body' => $csv
        ]);

Close

        $response = Forrest::jobs('ingest/' . $job['id'] . '/', [
            'method' => 'patch',
            'body' => [
                "state" => "UploadComplete"
            ]
        ]);

[omniphx]

Thanks @stevenwoodson for the detailed breakdown. I'll try to tinker with this a little. If you're ever interested in testing out your changes feel free to give a shot. I put together a guide on setting up a development environment: https://github.com/omniphx/forrest/blob/master/CONTRIBUTING.md

Happy to provide guidance as well. Lately been pre-occupied with other projects

[stevenwoodson]

I can verify that my change works great in my use case of sending a request with a CSV, going on the assumption that the header is modified to 'Content-Type' => 'text/csv'.

It seems logical that since this performs a json_encode that we just check for 'application/json' first before doing that, safe in the knowledge that that's the default Content-Type. But is that suitable for every use case? Do you have additional thoughts there?

[omniphx]

@stevenwoodson slightly modified your code for checking the default MIME type (most cases it will be application/json)

Any chance you could pull down the dev-master version and verify the change works for you?

[stevenwoodson]

Verified, so cool! I took a peek at the code updates and it all makes sense to me. Thanks for taking that on so quick!

[omniphx]

I'm gonna bump up the version. If you get a chance, I'd love some documentation on how to use the Bulk API

[stevenwoodson]

Sure will, I'm no expert but I can share what I've learned. I'll get something written up and start a PR for that soon.