The “project” argument was not found in the post data.

[rnovec]

Current documentation show examples to make request to Scrapyd API through curl. However, I am trying to make the same request in axios (Node.js) and getting this error:

Scrapyd response: { node_name: 'rnovelo-pc', status: 'error', message: "'project'" }

In terminal

File "/home/rnovelo/.local/lib/python3.6/site-packages/scrapyd/webservice.py", line 46, in render_POST
            project = args.pop('project')
        builtins.KeyError: 'project'

Is possible open issue to create OpenAPI or some standard documentation such as REST API? ... or what is the correct headers, encoding and/or things to consider to make request in other languages?

[my8100]

It seems that the “project” argument was not found in the post data. Maybe you should learn how to post the data with Python first (you can use the requests lib). Could you post the Node.js code?

[rnovec]

@my8100 I've already used python-scrapyd-api in Python and works fine!

The same with request (Node.js) but is deprecated.

Request

var request = require('request')
var options = {
  method: 'POST',
  url: 'http://localhost:6800/schedule.json',
  headers: {
    'Content-Type': 'application/json'
  },
  formData: {
    project: 'default',
    spider: 'sii-login',
    rut: '76909926',
    clave: 'Aparicio83'
  }
}
request(options, function (error, response) {
  if (error) throw new Error(error)
  console.log(response.body) // OK!
})

Axios

var axios = require('axios')

var config = {
  method: 'post',
  url: 'http://localhost:6800/schedule.json',
  headers: {
    'Content-Type': 'application/json'
  },
  data: {
    project: 'default',
    spider: 'my-spider',
    rut: '...',
    clave: '...'
  }
}

// fails
axios(config)
  .then(function (response) {
    console.log(response.data)
  })
  .catch(function (error) {
    console.log(error)
  })

[Digenis]

@rnovec try this https://superuser.com/questions/1261685/forward-a-tcp-connection-with-logging-using-socat and compare what is sent by request and by axios.

[danieldiazastudillo]

I'm trying to call a scrapyd schedule endpoint from a C# client (using HttpClient and RestSharp) with no success. Getting this same error. I think that scrapyd and twisted are not accepting async calls, maybe that's the reason why request works and axios fails...

Any ideas?

[jpmckinney]

I'm going to assume this is the same as #391 (we can continue there), where a CORS request typically sends an OPTIONS request first. When that request fails, I guess some libraries just silently fail and never make the follow-up POST request.