Support (out of the ordinary) - adding a proxy server doesn't work as expected

[quammy]

I have a server getting its internet connection from an ISP that blocks agent.syagent.com as an inappropriate website. Go figure. Battling this ISP to whitelist agent.syagent.com might be a too tedious work. Then another ISP might block. So I thought of putting my own proxy server - just receive the data and forward it. I adjusted the sh-agent.sh script to send the data to my proxy server and my server should just POST it 'AS IS' to agent.syagent.com

But something falls somewhere, as the SyAgent website doesn't show the data, even though the POST request ends successfully.

This is my NodeJS POST code, I hope you can advise me on why it fails:

const http = require('http')
const compression = require('compression')
const express = require('express')

const httpServerPort = process.env.SERVER_PORT

const app = express()
app.use(express.json({ limit: '50mb' }))
app.use(express.urlencoded({ extended: true, limit: '50mb' }))
app.use(compression())

const httpServer = http.createServer(app)
httpServer.listen(httpServerPort, () => {
    console.log(`HTTP server started on port ${httpServerPort}`)
})

app.post('/api/v1/syagent_proxy', async (req, res) => {
    // console.log(req.body)
    try {
        const postData = JSON.stringify(req.body)
        const options = {
            hostname: 'agent.syagent.com',
            port: 443,
            path: '/agent',
            method: 'POST',
            Headers: {
                'Content-Type': 'application/json'
            }
        }

        const innerReq = http.request(options, (innerRes) => {
            innerRes.on('data', (d) => {
                console.log(`statusCode: ${res.statusCode}`)    // CODE ARRIVES HERE AND PRINTS "statusCode: 200"
                res.sendStatus(200)
            });
        });

        innerReq.on('error', (error) => {
            console.error(error)
            res.sendStatus(500)
        });

        innerReq.write(postData)
        innerReq.end()

    } catch (err) {
        console.error(err)
        res.sendStatus(501)
    }
})

[Rashe]

You send JSON, the server expects form data :)

[quammy]

Yes, I saw the original data is JSON and thought this is the way to send. Anyway, thanks for your support! I changed to form data in this way:

const url = require("url")
const https = require('https')

app.post('/api/v1/syagent_proxy', async (req, res) => {
    // console.log(req.body)
    try {
        const postData = url.format({
            pathname: "/",
            query: req.body
        })
        const options = {
        hostname: 'agent.syagent.com',
            port: 443,
            path: '/agent',
            method: 'POST',
            Headers: {
                "Content-Type": "multipart/form-data",
                "Content-Length": Buffer.byteLength(postData)
            }
        }

        const innerReq = https.request(options, (innerRes) => {
            innerRes.on('data', (d) => {
                // console.log(d)
                console.log(`statusCode: ${res.statusCode}`)    // CODE ARRIVES HERE AND PRINTS "statusCode: 200"
                res.sendStatus(200)
            });
        });

        innerReq.on('error', (error) => {
            console.error(error)
            res.sendStatus(500)
        });

        innerReq.write(postData)
        innerReq.end()

    } catch (err) {
        console.error(err)
        res.sendStatus(501)
    }
})

But now I'm getting a timeout (from the script - timeout -s SIGKILL 30), while getting the 200 code. Same if I'm using application/x-www-form-urlencoded.

Any additional help will be most appreciated.

[quammy]

I decided to change the way and be as similar as I can to your original code, so I run wget from within the nodejs code with same value structure as being used in sy-agent.sh.

It works sometimes yes and other times no (sending manually for now, cron is disabled) and I'm always getting this error (from wget stderr):

2023-02-14T13:43:20.992434689Z stderr: Connecting to agent.syagent.com (188.114.96.7:443)
2023-02-14T13:43:20.992489063Z 
2023-02-14T13:43:45.994305298Z stderr: wget: download timed out
2023-02-14T13:43:45.994362277Z 
2023-02-14T13:44:21.125290795Z child process exited with code 1

For now I have not enabled the cron job as I don't want to load or do any harm to the website server.

This is the code, if required:

const url = require("url")
const { spawn } = require("child_process")

app.post('/api/v1/syagent_proxy', async (req, res) => {
    // console.log(req.body)
    try {
        const postData = url.format({
            pathname: "/",
            query: req.body
        })
        // console.log(postData.substring(2).replaceAll("%20"," "))
        const wget = spawn("wget", [
          // "-q",
          // "-o",
          // "/dev/null",
          // "-O",
          // "/etc/syAgent/test.log",
          "-T",
          "25",
          "--post-data",
          postData.substring(2).replaceAll("%20"," "),
          "--no-check-certificate",
          "https://agent.syagent.com/agent"
        ]);

        wget.stdout.on("data", data => {
          console.log(`stdout: ${data}`);
        });

        wget.stderr.on("data", data => {
          console.error(`stderr: ${data}`);
        });

        wget.on("close", code => {
          console.log(`child process exited with code ${code}`);
        });

    } catch (err) {
        console.error(err)
        res.sendStatus(501)
    }
    res.sendStatus(200)
})

[Rashe]

Hi, try to send the data once in a minute. Tell me if it works for you

[quammy]

So I enabled the cron job. It took some minutes but then the website started to reflect the data. Every now and then it loses the updates ("Data Since: X minutes ago") and then it returns to "seconds ago". The error stderr: wget: download timed out continues on every request. Is that a normal wget behavior?

[Rashe]

Not really, but I will check it.

[quammy]

So wget was not a god idea as it works in serial and blocks everything while it runs.

I now have a working version which is the following:

app.post('/api/v1/syagent_proxy', async (req, res) => {
    try {
        const postData = querystring.stringify(req.body)
        const options = {
            hostname: 'agent.syagent.com',
            port: 443,
            path: '/agent',
            method: 'POST',
            protocol: 'https:',
            headers: {
                "Content-Type": "application/x-www-form-urlencoded",
                "Content-Length": Buffer.byteLength(postData)
            }
        }

        const innerReq = https.request(options, (innerRes) => {
            innerRes.on('data', (d) => {
                res.sendStatus(200)
            });
        });

        innerReq.on('error', (error) => {
            console.error(error)
            res.sendStatus(500)
        });

        innerReq.write(postData)
        innerReq.end()

    } catch (err) {
        console.error(err)
        res.sendStatus(501)
    }
})

Thanks!