Giteabot account was compromised #4167

Closed lafriks closed 6 years ago

We are currently investigating suspiscious activity from an account with write access priviledge to go-gitea organization. A binary was added to releases across multiple go-gitea repositories. We cleaned up all releases and drop temporarily access from the account. We will investigate futhermore to understand what really happen to prevent it in the future and be transparent with you trough the process. In the meantime, if you find any suspicious activity please report them under this issue.

UPDATE: No source code or other Gitea infrastructure was affected, including https://dl.gitea.io/ so it's safe to use it to download Gitea binaries.

GitHub account that was hacked is under full control and also have set 2FA so this should not happen in future again.

What was done:

Most of go-gitea organization repositories new release&tag was created with name 0 and added install.exe binary (13KB in size) to that release that was malicious (from our analysis contained crypto currency miner). All these releases and binaries was deleted within 2-3 hours from when they were added.
Also 1.4.2 release windows Gitea .exe binary on GitHub was replaced by same 13K binary as above. So if Gitea is working, you are safe.
Just in case we did retag 1.4.2 to trigger CI to rebuild binaries so sha256 will be different now as it was before retag.

We have contacted GitHub but have not received any answer from them, yet

UPDATE2: No actual gitea binaries were compromised so all hashes mentioned in comments below are safe.

SHA256 of malicious .exe file: bfc5a0358b1ad76ffbc1e1f4670bd3240536e2fbac88272cee3003322a15fffe

UPDATE3: v1.4.2 has been rereleased at about 2018-06-07 20:00:00 UTC+08

In the meantime please be careful when downloading our released binaries, especially the windows ones, until further notice. E.g. keep an eye on the size of the binaries, or a double .exe file ending. We've found out our .exe binaries within release 1.4.2 were altered as well.

We work hard to follow all trails, clean up and get back to daily business as soon as possible.

@daviian Maybe then it is necessary to sign releases?

@Mauladen Thank you for your input. We definitely will discuss this.

Ouch! Yeah, signed binaries would be ideal.

default

@Mauladen We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones.

Or did you mean something else?

This is normal. We retaged 1.4.2 to retrigger CI to rebuild binaries as windows binary for that release was removed and there was added new one malicious one

@daviian, Maybe @Mauladen meant that 1.4.2 release commit without GPG signature, but 1.4.1 was

Still need to edit the list of changes

@axifive commit where not gpg signed by user but github that does it automaticaly (with github key) when the merger is the same the PR author. I wouldn't it consider more safer because if github account were compromised I will been also show as "verified". But we could start to sign tag from now (to be discuss). For information we are, working on signing binary since it is more easy to corrumpt thzan binary as the git commit tree.

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify. You can get an idea on how that works/looks here:

Libressl uses the same technology.

Is there anymore information on this? What was the payload of the binary? Are users going to be informed about this through a blog post or anything? Were any of the linux binaries compromised? I'm rather concerned about what these things may have done to my servers. I'm doubly concerned that I only found out about this browsing the issue page by chance vs an official notice on the project page/blog

Is the Gitea 1.4.2 release safe to update to from source code at this point in time?

At this point, it's not clear to me whether only binaries were affected. How did you verify this? Are your branches protected?

shasums differ on binaries I downloaded on 6/4 and 6/7 though they are the same number of bytes:

$ sha256sum gitea-1.4.2-linux-amd64*
e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9  gitea-1.4.2-linux-amd64
c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d  gitea-1.4.2-linux-amd64.1

$ ls --color=tty -l gitea-1.4.2-linux-amd64*
-rwxr-xr-x 1 matt matt 53674800 Jun  4 20:39 gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 matt matt 53674800 Jun  7 08:18 gitea-1.4.2-linux-amd64.1

Feel like hosting those somewhere so folks can tear them apart?

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA. I'll put them somewhere more permanent if necessary.

Question - is it just the binaries on the website that were affected, or were the repositories also affected? I ask because yesterday I heard about Gitea and cloned and built the v1.4 branch.

Would really like some more info if the source itself is compromised or just the binaries. I run a script to check for updates/build from git nightly and was lucky enough to miss this because of the timing.

Is the current 1.4.2 release verified to be clean? If we know that's tainted we should pull that ASAP and put it somewhere else for analysis, otherwise people will still be downloading that if they don't see this issue. I agree with @CountMurphy, can we add something to the README so it's really visible?

Can we get hashes so we can check if our binaries are affected? My gitea 1.4.1 (linux amd64) looks like this: $ sha256sum gitea d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 gitea

I just downloaded gitea 1.4.1 linux-amd-64 and I got d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851 as the sha256 too

@mikedilger See: https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229

To make it overwhelmingly clear: nobody knows anything and the only safe hashes are the ones currently found here: https://github.com/go-gitea/gitea/releases

For Gitea 1.4.2 on amd64, that means:

c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d

Sorry, I misunderstood https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229 to mean that both binaries were compromised.

I've edited this post to remove any ambiguities about what we actually know.

Hold on: according to this comment (https://github.com/go-gitea/gitea/issues/4167#issuecomment-395576229), there are two shas e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 4th and c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d from June 7th.

Are we saying that both of these are compromised or just one of them? For the record the one on my server is e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 from June 5th.

@christianbundy, Please, don't make hasty conclusions, wait for a response from the Team member

Please, don't make hasty conclusions, wait for a response from the Team member

Sorry about that, I thought the file hashes would be posted by a team member immediately and didn't realize that the info was coming from someone else who was also in the dark. Is this the correct channel to watch for the latest developments? I've powered off my server and need more information to tell whether I've been infected.

I see your edits crossing out the entry I pointed to. HOWEVER, isn't this too early to draw conclusion from? How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

We're still missing a couple pieces of information (likely non-exhaustive):

[ ] Which binaries were compromised and what are their checksums?
[ ] When did the bot account get compromised?
[ ] Is the source repository compromised (this might be easy to check if you have a version of the repo cloned from some while back, then you can maybe check diffs or evidence of force pushes).

I got:

# ls -la gitea-1.4.1-linux-amd64
-rwxr-xr-x 1 gitea gitea 53663640 May  3 08:02 gitea-1.4.1-linux-amd64

With sha256sum d8cfa0d39da70497f1f75e519e4fee33e5ee7c0de88919bdfe46a8b0d38af851

and I just downloaded gitea-1.4.2-linux-amd64:

# ls -la gitea-1.4.2-linux-amd64
-rwxr-xr-x 1 root root 53674800 Jun  7 14:18 gitea-1.4.2-linux-amd64

With sha256sum c843d462b5edb0d64572b148a0e814e41d069d196c3b3ee491449225e1c39d7d which matches the provided .sha256 file: gitea-1.4.2-linux-amd64: OK which should be safe. (right?)

[...] We've rebuilt the binaries for 1.4.2 release just to be sure we provide safe ones. [...]

I uploaded gitea-1.4.2-linux-amd64 for analysis here, but it doesn't really tell anything obvious.

Going to watch this issue and keep my gitea installation offline for the time being.

How do we know for sure e14e54f334ce022d2026d0801da1ed1bf3bcba751b16fb290bae4d10d14482e9 is fine?

@shuhaowu I said that c843d462 was fine, not e14e54f3. We know this because that's the file currently being served on GitHub, which they have recently rebuilt.

If 1.4.2 was rebuilt, that should be signed and verified like the other valid releases. Not doing so only adds to the confusion.

@xcolour

Uploaded the binaries here: https://send.firefox.com/download/a5dc9a9335/#_k9e4Cu2r4pPNDrsS1T2CA

The only difference between these two binaries is that about 2000 instances of movabsq $63663754793, %rcx were replaced with movabsq $63663969431, %rcx. I can't say exactly what that changes behaviorally, but I think it's very unlikely that it's enough to be malicious.

Maybe push out a new (signed?) 1.4.3 release with the known-safe code, or would that confuse things more?

@justinclift I like that idea, also a blog post and something in the README about the situation would help clear things up.

@spaghetti2514

On one hand, you're right -- the differences between the those two binaries is easy to see.

On the other hand, the Gitea team hasn't actually posted which binaries were affected, posted any SHA256 hashes, or really said anything that would help us understand the compromise As others have pointed out, we don't have nearly enough info to determine what happened and who's affected.

I'm frustrated to point out that we should probably just assume the worst and wait for diagnostic steps from the core team. With that said, I appreciate you posting the disassembled diff.

Is this what you're looking for? https://github.com/go-gitea/gitea/issues/4167#issuecomment-395579466

Thank you for the update @lafriks !

I have updated issue description with additional info. It was not so bad as it could have been. We wanted to be as transparent as possible so created this issue as soon as we did clean up & fixed everything and as this was first time something like this have happened, so probably we should have given more information faster, sorry for causing confusion.

@lafriks Thanks for the update! Could you post your PGP key that we can use going forward to verify commits?

@4oo4 tags are signed by one of mergers own GPG key, as PR's all merged using Squash&Merge, they are not signed (or maybe signed by GitHub's magic :) )

Releases will be signed by GPG key that will be created before 1.5.0 release, we will publish it on README.md, gitea docs and in blog post.

As a 386 user, I can confirm that the gitea-1.4.1-linux-386 binary currently available on the Releases page matches a version I downloaded on June 4th (cf6344b4).

as PR's all merged using Squash&Merge, they are not signed (or maybe signed by GitHub's magic :) )

Don't use the merge button, that's basically insecure and a random gpg key. Merge locally and push the merge.

@mappu gitea v1.4.1 has not been affected and now v1.4.2 has been rereleased.

You should upload a tarball created by git-archive (in addition to those that github automatically generates) which you sign with signify .

Actually, you do not even have to build the archives and upload it again. You can sign the one GitHub created, because it can be done reproducibly.

Here is a small script for that: https://github.com/rugk/gittools/blob/master/signrelease.sh

@rugk Useful looking script. :smile:

Has any contributing team member got an archive of 1.4.2 prior to breach discovery?

You seem to have left a lot of people in the dark on whether or not this binary has been altered for any Linux releases, which is quite an important piece of information considering:

1) Most deployments will be to Linux stacks

2) Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

While I'd like to believe that this was a simple Windows-targetted attack and nothing more, the fact that they targeted this specific repository just days into a dramatic spike in userbase growth seems indicative to a more well orchestrated effort. I highly doubt that anyone who knew what they were targeting and how to pull off this sort of thing would have just passed up such a ripe opportunity to infect as many hosts as possible.

@stanier dl.gitea.io was not affected and we have verified that no other binaries were tampered with

So that explains why our webproxy refused to download the latest gitea image from hub.docker.com ...

@lafriks So can you confirm that c843d462 and e14e54f3 are both safe? If the CI provided a build with a different signature, then something must have changed in either the source code or the parameters the compiler used for the build. It's a minor technicality by itself, but it was never directly stated here whether or not the adversary had altered the 1.4.2 Linux binaries, only the respective .exe binaries mentioned in the comment by @daviian.

Just to clarify, I'm asking about the GH repo's release page binaries, not dl.gitea.io, I understand nothing was tampered with outside of the GH repo.

Linux stacks are not accustomed to binary trojans and aren't typically suited with the best tools for programmatically detecting suspicious code already running on the system.

Hmmm, don't most package managers (and similar) have provision for verifying at least sha* checksums? Not necessarily just for detecting malware, but as a "lets verify the download wasn't corrupt".

@justinclift yes, but that only applies to binaries installed through the package manager. The issue in question here concerns with a direct download from GitHub, which doesn't have any embedded malware detection to the best of my knowledge.

go-gitea / gitea

Giteabot account was compromised #4167