Some issue with my gitea setup, not sure what. #40

Closed mannp closed 9 months ago

Hi there

Thanks for creating typhon :)

I have;

A typhon and gitea hosted on separate servers each behind a nginx proxy using nix on a local network, both with valid https.

A typhon config with secrets and a project repo on gitea as per your examples.

A gitea webhook with write repo permissions, which is failing with response 400 Bad request: Bad webhook output. https://<typhon server>/api/projects/typhon-go-local/webhook

Headers
Connection: keep-alive
Content-Length: 31
Content-Type: text/plain; charset=utf-8
Date: Fri, 23 Feb 2024 20:26:09 GMT
Server: nginx

If I try to go to https://typhon server/api/ it 404's, so I am not clear if;

1) my nginx configs are causing issues with typhon or Gitea 2) why https://typhon server/api/ 404's when https://typhon server/ works fine?

Apologies for all the questions but I thought you might be able to rule out some things, that might help me find the issue :)

The following typhon log shows the webhook failure and some other warnings?

Thanks

Feb 23 20:24:32 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[11944]: 2024-02-23T20:24:32.204623Z  WARN actix_session::middleware: The session cookie attached to the incoming request failed to pass cryptographic checks (signature verification/decryption).
Feb 23 20:26:09 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[11944]: 2024-02-23T20:26:09.191355Z  WARN typhon_core: bad webhook for project typhon-go-local
Feb 23 20:26:12 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[11944]: 2024-02-23T20:26:12.562610Z  WARN actix_session::middleware: The session cookie attached to the incoming request failed to pass cryptographic checks (signature verification/decryption).
Feb 23 20:26:14 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[11944]: 2024-02-23T20:26:14.028205Z  WARN actix_session::middleware: The session cookie attached to the incoming request failed to pass cryptographic checks (signature verification/decryption).
Feb 23 20:34:41 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[11944]: 2024-02-23T20:34:41.827954Z  WARN actix_session::middleware: The session cookie attached to the incoming request failed to pass cryptographic checks (signature verification/decryption).
Fe

First of all, thank you for trying Typhon!

I am almost certain that nginx has nothing to do with your issue. The "Bad webhook output" error indicates that the headers and the body of the webhook request were correctly passed to the webhook action by Typhon. So the failure is very probably due to an improper project declaration or a faulty gitea setting. This highlights two problems on our part:

in the webapp there is currently no way to access the logs of the webhook action
the response sent to the webhook sender should contain more details about what's actually happening

Now to fix your issue. I see two ways the webhook action can fail: either the body of the request is not proper json, or the authentication failed. Thus, there are two things to check right away:

is the content type of the request properly set to application/json in the gitea settings?
is the webhook secret properly set and stored as gitea_webhook_secret in your project's secrets.age?

They don't seem related, but I am a bit worried about the actix_session warnings though. I have not seen them while running Typhon locally. @W95Psp can you check if they appear in the logs of etna?

Thanks for coming back to me.

I am not at the machine atm, and will update with more details shortly.

Yes, I checked the type myself, as nginx was reporting a different type as above. It is set to application/json for sure.

The webhook secret is more sketchy for me as wasn't fully clear how to set it.

I created a webhook secret and added its value to the secret field of the Gitea webhook.

I wasn't clear how the Gitea token was being picked up either.

I looked in the typhon Gitea PR and used the token names from that PR in the secrets.json on the config repo.

Re actix_session, there are a lot, I only pasted a small number. I initially thought they're related to Gitea cookies, but my changes made no différance.

Request URL: https://typhon/api/projects/typhon-go-local/webhook
Request method: POST
Content-Type: application/json
X-Gitea-Delivery: 136c00c9-007d-451a-ad2f-76f9d56dab27
X-Gitea-Event: push
X-Gitea-Event-Type: push
X-Gitea-Signature: 1bd5b0a6594ccd3bb1456456456451d38e15b5f581211a871db09a7eddd62bf2

project repo typhon-project

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    typhon.url = "github:typhon-ci/typhon/pnm/lib-refactor";
  };

  outputs = inputs: {
    typhonJobs."x86_64-linux".hello = inputs.nixpkgs.legacyPackages."x86_64-linux".hello;
    typhonProject = inputs.typhon.lib.gitea.mkProject {
      instance = "typhon";
      owner = "mannp";
      repo = "typhon-project";
      secrets = ./secrets.age;
      typhonUrl = "https://typhon";
    };
  };
}

config repo typhon

{
  "gitea_token": "token_created_in_gitea",
  "gitea_webhook_secret": "token_added_to_the_secret_field_in_the_gitea_webhook"
}

Request URL: https://typhon/api/projects/typhon-go-local/webhook
Request method: POST
Content-Type: application/json
X-Gitea-Delivery: 136c00c9-007d-451a-ad2f-76f9d56dab27
X-Gitea-Event: push
X-Gitea-Event-Type: push
X-Gitea-Signature: 1bd5b0a6594ccd3bb1456456456451d38e15b5f581211a871db09a7eddd62bf2

Is that the full list of headers sent by gitea? If so, there is no chance the webhook action is going to run properly and this all might be my fault! The webhook action expects the following two headers:

X-Hub-Signature-256
X-GitHub-Event

I tested the support for gitea on https://codeberg.org/ which sends those headers. But codeberg actually runs forgejo, not gitea. As far as I know forgejo is, for the time being, a drop-in replacement to gitea. Plus, I specifically set the webhook type to gitea in the settings. So I am surprized to see such a discrepancy between the two :thinking:

No there were more, I cut them down, presumed wrongly that just the Gitea stuff was used....will post the full lot now.

Request URL: https://typhon/api/projects/typhon-go-local/webhook
Request method: POST
Content-Type: application/json
X-GitHub-Delivery: 4e13e474-fd0f-4bba-bb5e-e4dd4dafc81e
X-GitHub-Event: push
X-GitHub-Event-Type: push
X-Gitea-Delivery: 4e13e474-fd0f-4bba-bb5e-e4dd4dafc81e
X-Gitea-Event: push
X-Gitea-Event-Type: push
X-Gitea-Signature: 88a91898d57200c253ea9ba906b5a012915036e02f59f78079c2877537476dea
X-Gogs-Delivery: 4e13e474-fd0f-4bba-bb5e-e4dd4dafc81e
X-Gogs-Event: push
X-Gogs-Event-Type: push
X-Gogs-Signature: 88a91898d57200c253ea9ba906b5a012915036e02f59f78079c2877537476dea
X-Hub-Signature: sha1=72ac7c7a1b1384c3b8a7fd30b350a01cb78847df
X-Hub-Signature-256: sha256=88a91898d57200c253ea9ba906b5a012915036e02f59f78079c2877537476dea

Rebuilt typhon this morning and it's different from yesterday;

Feb 24 12:23:59 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[4676]: 2024-02-24T12:23:59.235124Z  INFO actix_server::builder: starting 8 workers
Feb 24 12:23:59 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[4676]: 2024-02-24T12:23:59.235185Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
Feb 24 13:03:59 tcan hyi1mglizdzbm7pnvf49mqr21yl9azdq-typhon-start[4676]: 2024-02-24T13:03:59.601441Z  WARN typhon_core: bad webhook for project typhon-go-local

Not sure why it's a WARN, when it's clearly a critical ERROR :)

Edit: Tried a few other things, inc new full access token, with no joy for the moment, except I note I am not getting a Locked URL on the Project. actix_session issues are also back.

Tried spinning up a similar config with Forgejo, and it has the same issue for me :-/

About the warn instead of the error: our semantic is that an error is something unexpected happening in Typhon, bugs in user-defined behaviours like actions or evaluations are kept as warnings. Ideally warnings should be things that are reported properly to the user throught the interface, which as you can see is not yet the case!

Were you getting a locked URL before you tried other things? nothing is going to work if there is no locked url! (maybe this is also something that should be clearer in the interface, with a warning icon if the locked URL is empty, cc @W95Psp)

Every time you edit your project's configuration you should press the refresh button, the icon next to it should become green and you should get a locked URL. If that's not the case, try running nix build .#typhonProject.actions.x86_64-linux in your config flake and see if the actions are building properly.

Thank you again for going through this, once we manage to fix the issue for you I will open a few issues to improve the initial setup of a project!

About the warn instead of the error: our semantic is that an error is something unexpected happening in Typhon, bugs in user-defined behaviours like actions or evaluations are kept as warnings. Ideally warnings should be things that are reported properly to the user throught the interface, which as you can see is not yet the case!

I see, okay.

Were you getting a locked URL before you tried other things? nothing is going to work if there is no locked url! (maybe this is also something that should be clearer in the interface, with a warning icon if the locked URL is empty, cc @W95Psp)

Mmh, I see, so pretty fundamental to anything else happing then :) ... it would be great to get some feedback at this point that without the lock, nothing will happen.

So no, no lock, only for your example url.

I have tried, private, public typhon-config/typhon-project repos, git+https and just https for the config url. Added a user and repo type to the url and no joy.

I use authentik in front of forgejo and not sure if these errors below are referring to that, but it also appears to be looking in 'master' and not 'main'?

Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]: 2024-02-27T13:36:30.463290Z  WARN typhon_core::projects: refresh error for project test: Nix error: Evaluation error: NixCommand {
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     cmd: "\"nix\" \"flake\" \"lock\" \"--output-lock-file\" \"/dev/stdout\" \"--override-input\" \"x\" \"git+https://forgejo/mannp/typhon-config/src/branch/main\" \"/nix/store/vakn94snphshm1s6lcmb8fg6aysnnh5k-typhon-flake\"",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     stdout: "",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     stderr: "\r\u{1b}[K\r\u{1b}[K\r\u{1b}[Kfatal: unable to update url base from redirection:\n  asked for: https://forgejo/mannp/typhon-config/src/branch/main/info/refs?service=git-upload-pack\n   redirect: https://forgejo/user/login\n\r\u{1b}[Kwarning: could not read HEAD ref from repo at 'https://forgejo/mannp/typhon-config/src/branch/main', using 'master'\n\r\u{1b}[Kfatal: unable to update url base from redirection:\n  asked for: https://forgejo/mannp/typhon-config/src/branch/main/info/refs?service=git-upload-pack\n   redirect: https://forgejo/user/login\n\r\u{1b}[Kerror:\n       … while updating the lock file of flake 'path:/nix/store/vakn94snphshm1s6lcmb8fg6aysnnh5k-typhon-flake?lastModified=1&narHash=sha256-LhCLis8UsNTo1O1cahDvcTH%2BUGYF%2BYvBhdlVIy%2BcDfA%3D'\n\n       … while updating the flake input 'x'\n\n       … while fetching the input 'git+https://forgejo/mannp/typhon-config/src/branch/main'\n\n       error: program 'git' failed with exit code 128\n",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]: }
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]: 2024-02-27T13:36:30.704585Z  WARN typhon_core::projects: refresh error for project test: Nix error: Evaluation error: NixCommand {
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     cmd: "\"nix\" \"flake\" \"lock\" \"--output-lock-file\" \"/dev/stdout\" \"--override-input\" \"x\" \"git+https://forgejo/mannp/typhon-config/src/branch/main\" \"/nix/store/vakn94snphshm1s6lcmb8fg6aysnnh5k-typhon-flake\"",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     stdout: "",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]:     stderr: "\r\u{1b}[K\r\u{1b}[K\r\u{1b}[Kfatal: unable to update url base from redirection:\n  asked for: https://forgejo/mannp/typhon-config/src/branch/main/info/refs?service=git-upload-pack\n   redirect: https://forgejo/user/login\n\r\u{1b}[Kwarning: could not read HEAD ref from repo at 'https://forgejo/mannp/typhon-config/src/branch/main', using 'master'\n\r\u{1b}[Kfatal: unable to update url base from redirection:\n  asked for: https://forgejo/mannp/typhon-config/src/branch/main/info/refs?service=git-upload-pack\n   redirect: https://forgejo/user/login\n\r\u{1b}[Kerror:\n       … while updating the lock file of flake 'path:/nix/store/vakn94snphshm1s6lcmb8fg6aysnnh5k-typhon-flake?lastModified=1&narHash=sha256-LhCLis8UsNTo1O1cahDvcTH%2BUGYF%2BYvBhdlVIy%2BcDfA%3D'\n\n       … while updating the flake input 'x'\n\n       … while fetching the input 'git+https://forgejo/mannp/typhon-config/src/branch/main'\n\n       error: program 'git' failed with exit code 128\n",
Feb 27 13:36:30 tcan k227qdbrcv4m23plxr1nspdjn7ighvds-typhon-start[3913]: }

Every time you edit your project's configuration you should press the refresh button, the icon next to it should become green and you should get a locked URL. If that's not the case, try running nix build .#typhonProject.actions.x86_64-linux in your config flake and see if the actions are building properly.

~/nixos-shared/typhon-config/result

lrwxrwxrwx     3 root root         61 Jan  1  1970 begin -> /nix/store/wak9yj9bf9lbvbwwg2wf7sc5b5ka578v-action/bin/action
lrwxrwxrwx     3 root root         61 Jan  1  1970 end -> /nix/store/wak9yj9bf9lbvbwwg2wf7sc5b5ka578v-action/bin/action
lrwxrwxrwx     2 root root         61 Jan  1  1970 jobsets -> /nix/store/73r85gln1isq4kkfb5s27hfqxmmg0n0s-action/bin/action
lrwxrwxrwx     2 root root         55 Jan  1  1970 secrets -> /nix/store/wgbsnxbn93792d52fzpnrgzxp38j28fp-secrets.age
lrwxrwxrwx     2 root root         61 Jan  1  1970 webhook -> /nix/store/4ppskgn9fmzx7a8y8kjjvxvwmyb50nf2-action/bin/action

It seems to me the actions are building fine locally? No errors with nix build .#typhonProject.actions.x86_64-linux

Also, I note in the webhook request content;

"ssh_url": "forgejo@forgejo:mannp/typhon-config.git"

While ssh into forgejo is disabled.....is that a significant?

Mmh, I see, so pretty fundamental to anything else happing then :) ... it would be great to get some feedback at this point that without the lock, nothing will happen.

Yes definitely, it is these kinds of things that seem obvious once you wrapped your head around the concepts, and you end up forgetting it really is not!

Is git+https://forgejo/mannp/typhon-config/src/branch/main the URL you entered for you config flake? Shouldn't this simply be git+https://forgejo/mannp/typhon-config? One way or the other it appears that the Nix build command at refresh time is failing to clone the repository.

Maybe try running nix build git+https://forgejo/mannp/typhon-config#typhonProject.actions.x86_64-linux locally!

Yes definitely, it is these kinds of things that seem obvious once you wrapped your head around the concepts, and you end up forgetting it really is not!

Yes fair enough, that is true and works both ways :) having the config as only a public repo seems fundamental too.

Is git+https://forgejo/mannp/typhon-config/src/branch/main the URL you entered for you config

I have entered multiple urls trying to get it to work, but I started with git+https://forgejo/mannp/typhon-config

Maybe try running nix build git+https://forgejo/mannp/typhon-config#typhonProject.actions.x86_64-linux locally!

This works fine.

I made the config repo public again, and it builds without input locally.

Still fails on typhon server :-/

This works fine.

What happens if you run it from Typhon server?

This works fine.

What happens if you run it from Typhon server?

Progress .... it asks me for the username and password and then builds fine.

Not clear on why it's asking for credentials for a public repo :-/

Not clear on why it's asking for credentials for a public repo :-/

I don't know either :( But from what you told me, this might really well be because of authentik rather than gitea/forgejo?

At least we have the issue cornered! You need to be able to build the public repo from the server for Typhon to refresh the project. Either by figuring out how to disable the authentication for public repositories, or by passing the necessary credentials to Nix via nix.conf.

By the way, if you intend to use Typhon for private repositories, keep in mind that you are going to need to configure Nix to be able to access them. We currently have no native support for private repositories, but we are putting some thought into allowing users to pass arguments to Nix, including credentials for private repositories.

I don't know either :( But from what you told me, this might really well be because of authentik rather than gitea/forgejo?

Seems more to do with forgejo and gitea defaults, as my codeberg account when creating a non-private repo still needs my credentials to clone locally, while your example does not.

Will investigate further and circle back around on this.

Thanks for your time and patience getting this far.

Re Private repos, definitely a good thing to have, and having it within Typhon would be ideal. I will read on how passing via nix.conf might work too.

By the way, if you intend to use Typhon for private repositories, keep in mind that you are going to need to configure Nix to be able to access them. We currently have no native support for private repositories, but we are putting some thought into allowing users to pass arguments to Nix, including credentials for private repositories.

So both typhon-config and typhon-project have to be public repos at the moment?

So both typhon-config and typhon-project have to be public repos at the moment?

They don't have to be, it's just that you need to do extra work in your server's nixos configuration to pass credentials to Nix.

Ideally these credentials should be set from a project's declaration, without the need for a system update, and without the other projects on the same server being able to access them.

So, for me a total noob to nix ci, in summary, in no particular order;

Meant to be constructive and not critical :)

- No page auto refreshes when adding projects or updating job sets.

- Would be great to have some feedback from the system in the gui, on warnings or errors.

- Refresh button should be on the left or top, to associate it with refreshing the config definition only. 

- Job set button can be disabled until the config has gone green, as it does nothing and was misleading for me.

- Dark mode

- Ability to delete projects

- Stock setup, typhon-config needs to be public.

- At which level should the webhook be created, top level, withing the typhon-config repo, within the typhon-project repo..... I created within the project repo level and it works now.

- Webhook appears to be optional for automation as once typhon configured correctly 'evaluate' can be pressed.

- Make sure for gitea/forgejo REQUIRE_SIGNIN_VIEW = false; # stops public repos working

- It would be nice to have the github repo and manual url on the typhon gui, for easy ref when debugging :)

- How do we update the project description, it doesn;t get updated in the example :)

Ideally these credentials should be set from a project's declaration, without the need for a system update, and without the other projects on the same server being able to access them

Makes sense, but no idea how to make that happen at the moment.

Also, the project already has the token, so I am unclear why the project repo would need to be public (without further config).

This is finally resolved (thanks for your patience) and can be closed :)

Nix evaluation status: Success

So, for me a total noob to nix ci, in summary, in no particular order;

Meant to be constructive and not critical :)

Thank you for your extensive feedback, and thank you for your patience! I will take the time to read your remarks thouroughly and open a few follow up issues for us to work on.

Makes sense, but no idea how to make that happen at the moment.

This requires editing your Nix configuration: https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-access-tokens

Also, the project already has the token, so I am unclear why the project repo would need to be public (without further config).

ATM secrets from the project are only used for actions. The evaluating/building process just calls Nix without any configurable arguments. That is definitely something we want to change!

Hooray indeed lol, some daft gitea/forgejo config setting overriding public repos was the main cause, that I clearly misunderstood its use.

Thank you for your extensive feedback, and thank you for your patience! I will take the time to read your remarks thouroughly and open a few follow up issues for us to work on.

Not really extensive, but a brain dump, as I appreciate when creating some assumptions are made regarding the users' knowledge, which was not the case for me lol.

This requires editing your Nix configuration: https://nixos.org/manual/nix/stable/command-ref/conf-file#conf-access-tokens

Yes I saw that in the manual, but wasn't clear how to link with the secret used by typhon, as not to confuse or duplicate things.

ATM secrets from the project are only used for actions. The evaluating/building process just calls Nix without any configurable arguments. That is definitely something we want to change!

That is useful to know now :)

I started a new issue with feedback, hope you don't mind, to keep it clear from my issue.

Yes I saw that in the manual, but wasn't clear how to link with the secret used by typhon, as not to confuse or duplicate things.

To be honest it seems impossible to do without duplication :/

I started a new issue with feedback, hope you don't mind, to keep it clear from my issue.

That's no problem at all, on the contrary!

ATM secrets from the project are only used for actions

Just realised, but isn't cloning a repo an action? :)

Just realised, but isn't cloning a repo an action? :)

The catch is that Typhon does not know about repositories! It only knows about flakes URLs. It is Nix that is responsible to fetch the repo (or the tarball, flake URLs don't necessarily link to git repositories!)

Just realised, but isn't cloning a repo an action? :)

The catch is that Typhon does not know about repositories! It only knows about flakes URLs. It is Nix that is responsible to fetch the repo (or the tarball, flake URLs don't necessarily link to git repositories!)

Now it fully makes sense :)

typhon-ci / typhon

Some issue with my gitea setup, not sure what. #40