Problem when sending files via http.post: stream ended unexpectedly

TimoRuetten commented 8 years ago

I am trying to use your service to convert my files into GeoJSON.

For this I am using the http.post module you can find here: https://www.npmjs.com/package/http-post

This is my full code for sending an existing file to your service to get the GeoJSON back:

http.post('http://ogre.adc4gis.com/convert', {

}, [
  {
    param: 'upload',
    path:'/existing/file.zip'
  }
], function(res){
  res.setEncoding('utf8');
  res.on('data', function(chunk) {
    console.log(chunk);
  });
});

And this is the result I am getting: {"error":true,"msg":"stream ended unexpectedly"}

Is there something I am missing ?

wavded commented 8 years ago

That project was last published 4 years ago which is a long time in Node land. We run all our tests using the superagent module. Another good one is the request module. I would give those a try first. Note, it must be a multipart form. See the tests for examples for superagent.

https://github.com/visionmedia/superagent https://github.com/request/request https://github.com/wavded/ogre/blob/master/test/api-test.js

TimoRuetten commented 8 years ago

Thanks you @wavded for your fast response.

I have changed my code and using now the request module. This looks good - because now I am not getting the error I have become before. But now - somehow - something else is happen and I am not getting debugging informations back.

I've tried 2 files.

2 times a zip file and 1 time a KMZ file.

All 3 files are working great when direct uploading on your website. But when using your API I am getting this problem:

With the first .zip file:

{"error":true,"msg":"invalid signature: 0x4b502f65"}

And with the KMZ and the other ZIP file:

<html><body><h1>504 Gateway Time-out</h1>
  The server didn't respond in time.
</body></html>

Without any more information. I've tested a absolut invalid file but then I will get a good error log back so I can see there is a problem with the file. The files I am sending you via POST are between 0.6KB - 10KB.

Is there a problem with your API, or a problem with me ? Sorry for these problems but I have no idea whats going wrong here ?

request.post({url: 'http://ogre.adc4gis.com/convert', formData: {
        upload: fs.createReadStream('/absolute/path/to/valid/file.zip')
      }}, (err, httpResponse, body) => {
        if (err) {
          return console.error('Upl failed:', err);
        }
        console.log('Server responded with:', body);
      });

wavded commented 8 years ago

Hmm, I just tried and didn't have any issues using request:

var request = require('request')
var fs = require('fs')

request.post({
  url: 'http://ogre.adc4gis.com/convert',
  formData: {upload: fs.createReadStream(__dirname + '/samples/sample.shp.zip')},
}, function(er, res, body) {
  if (er) return console.log(er)
  console.log(body)
})

What version of Node are you running? And I'm assuming you are on the latest request package.

TimoRuetten commented 8 years ago

I am using node 4.4.7 and request v2.74.0. I've also tried with superagent module like you've done in your tests. But I am getting the same responses.

What I am exactly doing is: Uploading the file complete to my server. After the file is uploaded completed I am using the absolute path to this file to attach it for the request / superagent module. I think this should work or I am wrong ?

wavded commented 8 years ago

It seems like what you are describing should work. I'm assuming the file is readable by your process? If you don't mind sharing I could try the failing zip file on my end and see what happens.

TimoRuetten commented 8 years ago

Edit: I've downloaded your sample.shp.zip file and tried to send it to your API. Same Problem: 504 Gateway Time-out.

So I think the problem is how I am sending the file to you. But: The file ist definitely existing so maybe I save it the wrong way on my server. But I am not doing else then to upload it to my server. Very strange! I will take a look at my function to check if I am doing something wrong.

TimoRuetten commented 8 years ago

Hey @wavded - I skipped now the part where I uplaoded the file first and everything (exactly) what I am doing now is:

  let fs = require('fs');
  let request = require('superagent');
  let file = process.env.PWD + '/public/sample.shp.zip';
  let notFound = false;
  try {
    let fileExist = fs.lstatSync(file);
  } catch(e) {
    console.log('File not found!');
    notFound = true;
  }
  if (notFound) return;
  request
  .post('http://ogre.adc4gis.com/convert')
  .attach('upload', file)
  .end((err, res)=>{
    if (err) {
      return console.error('Upl failed:', err);
    }
    console.log('Server responded with:', res);
  });

And the respond is:

{ [Error: cannot POST /convert (504)]
status: 504,
text: '<html><body><h1>504 Gateway Time-out</h1>\nThe server didn\'t respond in time.\n</body></html>\n',
method: 'POST',
path: '/convert' },

So: the file definitely existing, is not uploaded before and is send direct to your API.

wavded commented 8 years ago

Try again once. One of the servers in that cluster was stuck and perhaps that was the one you were being routed to.

TimoRuetten commented 8 years ago

I've tried again and had the same problem. But interesting what you are saying: Because when using the ogr2ogr2 module by my self I had the same problem. CPU was going to around 100% and it was an endless process which never ends when trying to convert the shapefile into geoJSON. Sadly without any debugging information.

Maybe it was my fault that you had that stuck ? I was sending your test shapefile (the code above) so nothing special ...

wavded commented 8 years ago

Hmm not sure there is a file with "SPC Tornado Reports" that is failing pretty regularly, not sure if that is you or not. Are you uploading a large file?

TimoRuetten commented 8 years ago

The last few uploads were your own test file. sample.shp.zip from your respository.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 3 years ago

This issue was closed because it has been stalled for 5 days with no activity.

wavded / ogre

Problem when sending files via http.post: stream ended unexpectedly #55