go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
45.21k stars 5.5k forks source link

Contribution system: History heatmap for user #6

Closed joubertredrat closed 6 years ago

joubertredrat commented 8 years ago

Hi guys,

I think that is a possible feature, a history heatmap similar to github or gitlab. Actually exists a plugin called Calendar HeatMap. I used this on mine project to heat application log and worked fine here. Then, is only a idea, what you think? :)

http://cal-heatmap.com/ https://github.com/wa0x6e/cal-heatmap

Reference: https://github.com/gogits/gogs/issues/1640

bkcsoft commented 8 years ago

I would prefer a solution that is in the backend, unless it's required to have it update without reloading. Unfortunately I can't seem to find anything that does that :unamused:

Also this would require caching, since it will fetch huge amounts of data from disk...

joubertredrat commented 8 years ago

Yes, this plugin build on front-end, with backend I don't know too, but we can consider make component for this.

In my case I use ajax to get data, but build on frontend anyway

xinity commented 8 years ago

following @bkcsoft retention strategy in cache is a must if we don't want gitea to waste ressources. something like in the latest 15days could be enough don't you think ?

bkcsoft commented 8 years ago

@xinity a simple LRU-cache should suffice in my opinion. Might be useful to be able to set the size of that cache though.
For my own projects I use hashicorp/golang-lru which works just fine and has a configurable size :slightly_smiling_face:

charlesreid1 commented 6 years ago

d3-calendar illustrates this really nicely and simply with javascript: https://bl.ocks.org/mbostock/4063318

git log is really flexible, so you can use it to output directly to JSON format, filter on users or dates, customize output format, etc (even has its own set of shortcodes). After an aggregation step, that data can be loaded into D3 calendar (basically just json objects with a key of the date and a value of the number of commits).

Here is a single-user activity graph that illustrates embedding the Javascript visualization in the home template file: git.charlesreid1.com and modified home template.

lunny commented 6 years ago

@charlesreid1 could you send a PR to gitea with your heatmap?

charlesreid1 commented 6 years ago

I'm happy to put together a visualization in an PR, but a couple of things need to be worked out first:

  1. Placement. My example heatmap is commit data scraped for a single user, displayed on the home page for the (single-user) gitea server - which doesn't make sense for multi-user gitea servers. If we wanted to have a per-user heatmap, it could go a couple of places:

    • Behind the "Public Activity" tab on the user's profile (most appropriate place, but not very visible)
    • Just on top of the three tabs "Repositories"/"Public Activity"/"Starred repositories" on the user's profile (most visible, on user profile so useful for others to see your activity graph)
    • On the top of the user's dashboard (less useful since others cannot see your activity graph)
  2. Data. The model I'm using for data would need to be changed a bit as well (my visualization points to a JSON file not dynamically generated by gitea). We could modify gitea to cache per-user commit data (updated with a post-commit hook or a daily maintenance task) and provide an endpoint route, where you provide it a username and it returns commit data in a JSON form convenient for visualization.

Thoughts? Potential roadblocks?

lunny commented 6 years ago

I would like put it under the Public Activity tab and top of all the activities on user's profile. Like the below example image: heatmap

joubertredrat commented 6 years ago

@lunny I liked this approach, if possible to change color to green can be best.... or not, github is green too.

charlesreid1 commented 6 years ago

Thanks @lunny for the suggestion, looks like a great spot for it.

@joubertredrat I'll do something plain and let the hipsters decide what color scheme to use. 🤓

lafriks commented 6 years ago

Green would be more consistent with gitea theme

charlesreid1 commented 6 years ago

I will use ColorBrewer's YlGn theme, which looks like the same one Github uses and is nicely consistent with Gitea green.

YlGn scheme

charlesreid1 commented 6 years ago

It looks like gitea/models/graph.go may already implement a model for commit counts. Is that correct?

charlesreid1 commented 6 years ago

Maybe not... But it looks like graph.go might be useful for implementing commit counts. There are a couple of files referring to a commit graph. Is this related to the heatmap/calendar idea, or is this referring to the timeline that shows up on a user's/organization's page?

lunny commented 6 years ago

I think we need a new table to store the statistics information.

type UserHeatMap struct {
      ID int64
      UserID int64 `xorm:"index"`
      Day string `xorm:"char(10)"` // 2018-02-26
      Count int
}

And all needed data could read from table action.

thehowl commented 6 years ago

Could we not use a SQL DATE instead of a char(10)?

lafriks commented 6 years ago

Also if this is just activity from action table than probably no new table is needed just select user_id, date, count(1) with group by user_id, date?

lunny commented 6 years ago

@lafriks that maybe select date, count(1) from action where user_id = ? and date >= ? and date < ? order by date group by date

lafriks commented 6 years ago

@lunny yes that what I meant but I don't know if xorm supports such queries

lunny commented 6 years ago
type Stat struct {
      Date string
      Count int
}
var stats = make([]Stat,0, 366)
x.SQL(`select date, count(1) as count from action where user_id = ? and date >= ? and date < ? order by date group by date`, 
userID, startDate, endDate).Find(&stats)
thehowl commented 6 years ago

would that work in all dialects?

lafriks commented 6 years ago

If we have date column (do we?) than it should work

lunny commented 6 years ago

@lafriks we can use created and remove the time parts.

lafriks commented 6 years ago

@lunny don't think removing time part will be that easy so that it would perform good enaugh for such select

carina-akaia commented 6 years ago

It's possible to see this feature in near future?

charlesreid1 commented 6 years ago

This effort was over my head, as there many decisions where I didn't have a clear sense of how to proceed, so I didn't get very far.

kolaente commented 6 years ago

Any updates on this?

lunny commented 6 years ago

@kolaente no people are working on this.

kolaente commented 6 years ago

Someone created a nice article on how to implement this externally: https://medium.com/@markuman/gitea-contribution-heatmap-af4be1d7a0d5

kolaente commented 6 years ago

I'll try to implement this.

lafriks commented 6 years ago

Problem is with this part:

group by DATE_FORMAT(FROM_UNIXTIME(created_unix), '%Y%m%d')

At least I did not find a way to implement it for sqlite

kolaente commented 6 years ago

@lafriks I did find one:

SELECT created_unix as timestamp, count(user_id) as contributions
FROM `action`
WHERE (user_id = 1)
  AND (created_unix > 1508503562)
GROUP BY strftime('%Y-%m-%d', created_unix, 'unixepoch')
ORDER BY created_unix

I think I'll do it with a hard switch to provide another query if we are dealing with an sqlite db.

lafriks commented 6 years ago

I think you will need different for mssql and Postgres also

kolaente commented 6 years ago

PR is up: #5131

naiba commented 6 years ago

Awesome, can't wait for release this.