Add support for OneDev devops platform

[KN4CK3R]

fixes #422

This PR adds support for OneDev. As OneDev lacks some needed API endpoints this Pull Request needs to merged first:

• [x] https://code.onedev.io/projects/onedev-server/pulls/28 (merged)

[alexwaibel]

Thank you for taking up this feature. Could you please add some test cases to cover this new integration? I would also appreciate if you could create a PR on the documentation website repo to cover the new config options and link to it here in this PR.

[KN4CK3R]

I hope I used the correct repository for the documentation: https://github.com/eduardoboucas/staticman.net/pull/28 Your link leads to a Github app?

[alexwaibel]

I hope I used the correct repository for the documentation: https://github.com/eduardoboucas/staticman.net/pull/28

Your link leads to a Github app?

Sorry, I linked from mobile and GH must redirect to the app. This is indeed the correct repo. I'll review but wait to approve the docs change till this PR is also ready.

[arthef]

Hi,

I am trying to test 1dev support. It is not clear to me however if the implementation supports setting 1dev through env variables. Documentation suggest that setting through config file is only supported using onedevBaseUrl and so on. Would these work if I set these variables as ENV variables?

This is essential to run Staticman from Docker.

[KN4CK3R]

The ENV variables are ONEDEV_BASE_URL, ONEDEV_USERNAME and ONEDEV_PASSWORD.

[arthef]

Also, there is staticman section in project's _config.yml file. What should be setting for endpoint for OneDev?

[KN4CK3R]

Have a look at the documentation: https://github.com/eduardoboucas/staticman.net/pull/28/files The default endpoint is https://code.onedev.io/api but you should use your instance.

[KN4CK3R]

I have added the suggestions and mentioned the env variables in the doc PR.

[arthef]

Thank you for relying but I was asking about Staticman endpoint configuration in _config.yml file.

• For GitHub it is: https://{your Staticman v3 API}/v3/entry/github/
• For GitLab this is: https://{your Staticman v3 API}/v3/entry/gitlab/

So, is there a corresponding https://{your Staticman v3 API}/v3/entry/onedev/ endpoint?

[arthef]

Looks like there is onedev endpoint after all.

However, and unfortunately I cannot get it working. After switching my Staticman setup from GitLab (working) to my own OneDev installation it no longer works.

I am trying to test it from command line using curl:

curl -X POST http://staticman.host:8880/v3/entry/onedev/kobit/myrepo/master/comments \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-raw 'fields%5Bname%5D=Richard&fields%5Bmessage%5D=This+is+a+test+comment.&fields%5Bemail%5D=user@domain.com'

Such a call works with my GitLab setup, however, when I change setup to my OneDev installation I am consistently getting the error:

{"success":false,"rawError":"RequestError: connect ECONNREFUSED 127.0.0.1:443"}

I spent some time to dig into this and found lots of references related to a similar error coming from using got package. Like this: https://forums.meteor.com/t/solved-problems-with-using-the-got-http-package/54297

There is more on the internet.

Looking at Staticman source code I can see that implementation for GitHub and GitLab do not use got package. This is only used by OneDev implementation. Hence the problem happens only for OneDev.

Unfortunately I have no idea on how to fix it or even implement some workaround.

So any hints on how to get it working would be very much appreciated.

I can provide more details on my environment if necessary.

[KN4CK3R]

Does your OneDev instance use https? I used OneDev in a docker container with http for my tests.

[arthef]

Yes, my OneDev runs over HTTPS. It is a bit more complex though. I run OneDev in Docker as well with plain HTTP but I also have nginx proxy manager in front which takes care of SSL and certificates.

[arthef]

I reconfigured Staticman to have it connect directly to 1dev over http. This time I got a different error:

{"success":false,"rawError":"HTTPError: Response code 404 (Not Found)"}%

However, the error does not have any details, therefore I have no idea what was not found.

[KN4CK3R]

I have tested it again with https://code.onedev.io and got the same ECONNREFUSED error. I played with the dependencies and the mailgun-js dependency is the reason for the error. It uses proxy-agent which eventually uses agent-base which causes the error.

Workaround:

const http = require('http');
const https = require('https');
const urlToOptions = require('got/dist/source/core/utils/url-to-options').default;

export default class OneDev extends GitService {
  constructor(options = {}) {
    super(options.username, options.repository, options.branch);

    this.api = got.extend({
      prefixUrl: config.get('onedevBaseUrl'),
      username: config.get('onedevUsername'),
      password: config.get('onedevPassword'),
      responseType: 'json',
      request: (url, options, callback) => {
        if (url.protocol === 'https:') {
            return https.request({...options, ...urlToOptions(url)}, callback);
        }
        return http.request({...options, ...urlToOptions(url)}, callback);
      }
    });
  }

Or migrate from mailgun-js to the official mailgun.js library. //cc @alexwaibel

[KN4CK3R]

For your 404 error: You need to specify the repository id instead of the repository name in the url. The OneDev api sadly does not use the names. Here is how I tested it:

1. Create a repository in OneDev: https://code.onedev.io/projects/test-staticman
2. Push the staticman.yml
3. Get the id of the repository: I don't know if the id is visible in the ui. I used the /api/projects endpoint which lists all projects. In this case 250.
4. Create the staticman config:

   {
   "onedevBaseUrl": "https://code.onedev.io/api",
   "onedevUsername": "KN4CK3R",
   "onedevPassword": "password or token",
   "rsaPrivateKey": "-----BEGIN RSA PRIVATE KEY---- ... -----END RSA PRIVATE KEY-----",
   "port": 3000
   }

5. Execute a POST request to http://localhost:3000/v3/entry/onedev/kn4ck3r/250/master/comments The username is not needed and can be anything. 250 is the project id.

Result: https://code.onedev.io/projects/test-staticman/pulls/3

[KN4CK3R]

I created a second PR to replace the mailgun-js dependency: #431

[arthef]

Hi,

I merged the PR #431 into my staticman repository: https://1dev.just-4.dev/projects/staticman

Both errors went away but I am not sure if I moved further because now staticman fails with the following error:

Successfully compiled 26 files with Babel (962ms).
Staticman API running on port 8880
(node:561797) UnhandledPromiseRejectionWarning: TypeError: _got.default.extend is not a function
    at new OneDev (/home/kobit/projects/staticman/build/lib/OneDev.js:23:29)
    at _default (/home/kobit/projects/staticman/build/lib/GitServiceFactory.js:22:14)
    at /home/kobit/projects/staticman/build/lib/Staticman.js:60:55
    at new Staticman (/home/kobit/projects/staticman/build/lib/Staticman.js:71:7)
    at _default (/home/kobit/projects/staticman/build/controllers/process.js:141:27)
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:244:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:223:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at /home/kobit/projects/staticman/build/server.js:208:14
    at Layer.handle [as handle_request] (/home/kobit/projects/staticman/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/kobit/projects/staticman/node_modules/express/lib/router/route.js:137:13)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:142:36)
    at module.exports.MemoryStore.set (/home/kobit/projects/staticman/node_modules/express-brute/lib/MemoryStore.js:28:35)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:127:17)
    at module.exports.MemoryStore.get (/home/kobit/projects/staticman/node_modules/express-brute/lib/MemoryStore.js:38:35)
    at module.exports.<anonymous> (/home/kobit/projects/staticman/node_modules/express-brute/index.js:82:15)
    at keyFunc (/home/kobit/projects/staticman/node_modules/express-brute/index.js:39:41)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:561797) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:561797) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

And I am running the staticman from a command line to test it like this:

NODE_ENV=production npm start

My config/production.json:

{
  "onedevPassword": "onedev-user-token",
  "onedevBaseUrl": "https://1dev.just-4.dev/api",
  "onedevUsername": "kobit",
  "rsaPrivateKey": "-----BEGIN RSA PRIVATE KEY----- key -----END RSA PRIVATE KEY-----",
  "port": 8880
}

I would appreciate if you have any suggestions.

[KN4CK3R]

Somehow you are missing got in your dependencies. Run npm i got. After that it worked for me again. And you are not using the latest commit from this PR.

[arthef]

Thank you. With suggestions from your last comment I was able to run it successfully from command line and it all worked.

However, it seems like applying all changes from both PR results in Staticman not working within Docker container.

As Staticman compiles source files at startup time, it needs development dependencies which are not automatically available. It seems that older version of node and modules might have worked differently.

Also, the current version requires that configuration file exists, even if it is empty, otherwise it will not start.

So, this is something the project maintainer has to be aware of, that accepting this PR will break the docker installation.

I managed to get it working with the following Dockerfile and a minor change in packages.json file to put all development dependencies into the production dependencies. It feels more like a workaround rather than a proper solution.

In any case, it is all working for me. If there is anything I need to do to release the bounty for the implementation, please let me know.

[KN4CK3R]

Hm, strange. Both PRs do not change how Staticman should be build but I haven't tested that.

In any case, it is all working for me. If there is anything I need to do to release the bounty for the implementation, please let me know.

You just need to close your issue #422. 👍

[KN4CK3R]

I see the problem but it's not from my PRs. The start command was changed in #396. It starts a build now instead of just executing: https://github.com/eduardoboucas/staticman/commit/ba5b96dfabfb5c436113f72de3422e532f59f7fd#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R11

[arthef]

Ok, issue is closed.

[alexwaibel]

There's one additional docs change I'd like before we merge this PR. I want to merge both at the same time. https://github.com/eduardoboucas/staticman.net/pull/28#pullrequestreview-741055130

I merged the mailgun dependency upgrade, but now there's a merge conflict on this PR that needs resolved as well. Really close to getting this merged, thanks.