Gitea Oauth2

[ambitslix]

Hello,

I have buildbot master in virtualenv:

Buildbot version: 2.7.0 Twisted version: 20.3.0

I installed buildbot_gitea like so:

pip install buildbot_gitea

I'm trying to set up gitea Oauth2, I have this in master.cfg from the README:

from buildbot.plugins import *
...
c['services'] = [
    # Report status back to gitea, verbose flag enables verbose output in logging for debugging
    reporters.GiteaStatusPush(
...
]
...

However I'm getting this:

2020-04-03 11:57:58-0700 [-] error while parsing config file:
    Traceback (most recent call last):
      File "/sandbox/lib64/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
        inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
      File "/sandbox/lib64/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/sandbox/lib64/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
        return func(*args,**kw)
      File "/sandbox/lib64/python3.7/site-packages/buildbot/config.py", line 170, in loadConfig
        self.basedir, self.configFileName)
    --- <exception caught here> ---
      File "/sandbox/lib64/python3.7/site-packages/buildbot/config.py", line 128, in loadConfigDict
        execfile(filename, localDict)
      File "/sandbox/lib64/python3.7/site-packages/twisted/python/compat.py", line 247, in execfile
        exec(code, globals, locals)
      File "/buildbot-master/master/master.cfg", line 121, in <module>

    builtins.NameError: name 'GiteaStatusPush' is not defined

I tried the following in master.cfg:

from buildbot_gitea import reporter
c['services'] = [
    # Report status back to gitea, verbose flag enables verbose output in logging for debugging
    reporter.GiteaStatusPush(
...
from buildbot_gitea import step_source

But now I'm getting:

2020-04-03 12:22:54-0700 [-] beginning configuration update
2020-04-03 12:22:54-0700 [-] Loading configuration from /buildbot-master/master/master.cfg'
2020-04-03 12:22:54-0700 [-] neither txrequests nor treq is installed, but GiteaStatusPush is requiring it

    txrequests is based on requests and is probably a bit more mature, but it requires threads to run,
    so has more overhead.
    treq is better integrated in twisted and is more and more feature equivalent

    txrequests is 2.8x slower than treq due to the use of threads.

    http://treq.readthedocs.io/en/latest/#feature-parity-w-requests
    pip install txrequests
        or
    pip install treq

I have done this then:

pip install treq txrequests

Stil the same error.

[pampersrocker]

I'm using treq for the requests provider as it was the only one working for an older gitlab integration. Not sure what happens if you have both installed. Try uninstalling txrequests (pip uninstall txrequests) and see if the problem persists.

I'm running buildbot 2.3.1 and do not have any problems using from buildbot.plugins import * and then using reporters.GiteaStatusPush or util.GiteaAuth.

[ambitslix]

Hello,

I've tried that already, but it's the same error. I suspect that the issue is the newer version of buildbot and that buildbot_gitea isn't working with it. The official version in my distro is 2.5.x I will try that, but considering that this is python they should work with any version.

[pampersrocker]

I may be able to upgrade our buildbot installation or at least try a newer buildbot version in another vm in the following days.

[ambitslix]

That would be great, I tested v2.3.1 in a virtualenv and it works with that. I'm not sure if I really need the latest version but having it work with gitea is ideal.

[pampersrocker]

I have not been able to reproduce this issue yet, On Python 3.5 (the newest available on the Debian System I used). Both of my plugins (the other being the https://github.com/pampersrocker/buildbot-UnrealEngine plugin) are recognized by the plugin import of buildbot. There seems to be something wrong in the entry point registration, but this is beyond the implementation of this plugin (see setup.py for the defined entry points, this is how the plugin registers itself with buildbot).

You may want to create an issue in the buildbot repository and see if they have an idea on why plugins are not registering. I've checked the changelog vom 2.3.1 onwards and did not see anything which could break this.

[ambitslix]

Thanks for looking into that. I have installed the latest via pip and it works, although it does complain about treq/txrequests but eventually after upgrading and installing missing plugins it works. The buildbot docs don't say how to install master with the required packages and plugins, there is a lot more then just pip install buildbot. Also some end up being the wrong version such as pyasn1 then needs to be manually upgraded.

6 pip install buildbot --user 8 pip install service_identity --user 10 pip install --upgrade pyasn1 --user 11 pip install --upgrade service_identity --user 13 pip install buildbot.www --user 20 pip install buildbot.gitea --user 25 pip install request --user 26 pip install --upgrade request --user 28 pip install --upgrade treq --user 31 pip install --upgrade buildbot.waterfallview --user 32 pip install --upgrade buildbot-waterfall-view --user 33 pip install --upgrade buildbot-www --user 37 pip install --upgrade buildbot-console-view --user 40 pip install --upgrade buildbot-grid-view --user

I know it's a different issue but I'm seeing this after adding to my master.cfg from here and adding webhook to one of my gitea projects:

The dialect specified, 'gitea', wasn't whitelisted in change_hook 2020-04-10 11:18:06-0700 [_GenericHTTPChannelProtocol,0,127.0.0.1] Note: if dialect is 'base' then it's possible your URL is malformed and we didn't regex it properly 2020-04-10 11:20:36-0700 [_GenericHTTPChannelProtocol,1,127.0.0.1] The dialect specified, 'gitea', wasn't whitelisted in change_hook 2020-04-10 11:20:36-0700 [_GenericHTTPChannelProtocol,1,127.0.0.1] Note: if dialect is 'base' then it's possible your URL is malformed and we didn't regex it properly

My payload URL is like this: https://domain.org/bb/change_hook/gitea (It's behind Nginx proxy) Perhaps I need to add something to whitelist gitea hook?

[pampersrocker]

The issue gets generated here: https://github.com/buildbot/buildbot/blob/117b63affb14a8d87cd18554b889d892647db6a7/master/buildbot/www/change_hook.py#L113

This sounds like you did not register the change hook handler, like described in the readme:

c = BuildbotMasterConfig = {}
c['www'] = {
    'change_hook_dialects': {
        'gitea': {
            'secret': '<SecretToEnterInGitea>',
            'onlyIncludePushCommit': True
        }
    },
}

[ambitslix]

I did it's there, something else is wrong...I have the master.sample.cfg modified and working and I inserted the example from here with my mods, so it's all there. Not sure what's wrong. Do I have to add to change_sources something? Here is my master.cfg(I know it's incomplete but I don't see why the change_hook would not work...I have to learn more about buildbot hooks probably):

cat master/master.cfg
# -*- python -*-
# ex: set filetype=python:

from buildbot.plugins import *

# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### WORKERS

# The 'workers' list defines the set of recognized workers. Each element is
# a Worker object, specifying a unique worker name and password.  The same
# worker name and password must be configured on the worker.
c['workers'] = [worker.Worker("example-worker", "pass")]

# 'protocols' contains information about protocols which master will use for
# communicating with workers. You must define at least 'port' option that workers
# could connect to your master with this protocol.
# 'port' must match the value configured into the workers (with their
# --master option)
c['protocols'] = {'pb': {'port': PORT}}

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.  Here we point to the buildbot version of a python hello-world project.

c['change_source'] = []
c['change_source'].append(changes.GitPoller(
        'git://github.com/buildbot/hello-world.git',
        workdir='gitpoller-workdir', branch='master',
        pollInterval=300))

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In this
# case, just kick off a 'runtests' build

c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
                            name="all",
                            change_filter=util.ChangeFilter(branch='master'),
                            treeStableTimer=None,
                            builderNames=["runtests"]))
c['schedulers'].append(schedulers.ForceScheduler(
                            name="force",
                            builderNames=["runtests"]))

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which workers can execute them.  Note that any particular build will
# only take place on one worker.

factory = util.BuildFactory()
# check out the source
factory.addStep(steps.Git(repourl='git://github.com/buildbot/hello-world.git', mode='incremental'))
# run the tests (note that this will require that 'trial' is installed)
factory.addStep(steps.ShellCommand(command=["trial", "hello"],
                                   env={"PYTHONPATH": "."}))

factory.addStep(steps.Gitea(
    repourl="ssh://git@domain.org:PORT/USER/test.git",
    mode='incremental',
    workdir="build",
    branch="master",
    codebase='test',
    progress=True,
    logEnviron=False,
))

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="runtests",
      workernames=["example-worker"],
      factory=factory))

####### BUILDBOT SERVICES

# 'services' is a list of BuildbotService items like reporter targets. The
# status of each build will be pushed to these targets. buildbot/reporters/*.py
# has a variety to choose from, like IRC bots.

c['services'] = []
c['services'] = [
    # Report status back to gitea, verbose flag enables verbose output in logging for debugging
    reporters.GiteaStatusPush(
        'https://domain.org/git', "xxxxxxxxxxxxxxxxxxxxxxx", verbose=True)
]

####### PROJECT IDENTITY

# the 'title' string will appear at the top of this buildbot installation's
# home pages (linked to the 'titleURL').

c['title'] = "My CI"
c['titleURL'] = "http://domain.org/index.html"

# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server is visible. This typically uses the port number set in
# the 'www' entry below, but with an externally-visible host name which the
# buildbot cannot figure out without some help.

c['buildbotURL'] = "https://domain.org/ci/"

c['www'] = {
    'change_hook_dialects': {
        'gitea': {
            'secret': 'SECRET',
            'onlyIncludePushCommit': True
        }
    },
}

# minimalistic config to activate new web UI
c['www'] = dict(port=PORT,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))

c['www']['authz'] = util.Authz(
        allowRules = [
            util.AnyEndpointMatcher(role="admins")
        ],
        roleMatchers = [
            util.RolesFromUsername(roles=['admins'], usernames=['USER'])
        ]
)
c['www']['auth'] = util.UserPasswordAuth([('USER','PASS')])

####### DB URL

c['db'] = {
    # This specifies what database buildbot uses to store its state.
    # It's easy to start with sqlite, but it's recommended to switch to a dedicated
    # database, such as PostgreSQL or MySQL, for use in production environments.
    # http://docs.buildbot.net/current/manual/configuration/global.html#database-specification
    'db_url' : "sqlite:///state.sqlite",

[pampersrocker]

c['www'] = {
    'change_hook_dialects': {
        'gitea': {
            'secret': 'SECRET',
            'onlyIncludePushCommit': True
        }
    },
}

# This assign c['www'] a new dict, overrding the one above
c['www'] = dict(port=PORT,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))

you should either append the dict or create it in one. Your code above overrides the dict with a new one.

e.g. something like

c['www'] = {
    'change_hook_dialects': {
        'gitea': {
            'secret': 'SECRET',
            'onlyIncludePushCommit': True
        }
    },
    'port': PORT,
    'plugins': {
        waterfall_view:{},
        console_view={},
        grid_view={}
    }
}

[ambitslix]

Thanks, right...there goes my python skills...:)

However now nothing appears in the log when I do a test hook deliver from gitea. Perhaps that's good but I doubt it...

c['www'] = {
    'change_hook_dialects':{
        'gitea':
        {
            'secret': 'SECRET',
            'onlyIncludePushCommit': True
        }
    },
    'port': PORT,
    'plugins':{
        'waterfall_view':{},
        'console_view':{},
        'grid_view':{}
    }
}

Ok I checked the gitea logs and the secret was mismatched, it works now. Thank you for all the help!