MVP repository group content curation

christinaforney commented 4 years ago

List to curate (check off when json configuation/content is complete below):

Collections we are adding new repogroups for -->

[x] Python 2 to Python 3 -> Top 100 Python libraries?
[x] React Hooks
[x] Android
[x] Top 100 Kubernetes OR group of container technologies from https://www.cncf.io/projects/?
[x] Go

Collections we already have repogroups and examples for-->
[ ] Rust crates
[ ] NPM
[ ] Angular
[ ] ethereum

Next iterations:

[ ] Deep Learning
[ ] C
[ ] C#
[ ] C++
[ ] Kubernetes
[ ] Go team
[ ] ethereum
[ ] Lua
[ ] Haskell
[ ] Stripe API
[ ] Swift
[ ] Yarn

Here is the JSON that contains our starting configuration and list of repository groups:

    {
      "id": "python2to3",
      "image": "TODO",
      "name": "Python 2 to 3",
      "value": "repogroup:Python 2 to 3"
      "short_description": "Required for card display on home and repository group page. Character limit is 110 characters.",
      "description": "Top 50 Github Python repos by number of stars.
            Search examples:
                 Python 2 style imports: "
                           repogroup: pythong2to3: from \w+ import \w+ 
                 Python 3 style imports:
                           repogroup: pythong2to3: from \B\.\w+ import \w+
                Python 2 style print statements,
                           repogroup: pythong2to3: print \":[string]\" lang:python
                 Python 3 style print statements:
                           repogroup: pythong2to3: print(\":[string]\")" lang:python
                 Python 2 style float division with type-conversion: 
                          repogroup: pythong2to3: float(:[_]) / float(:[_])
           ",
      "repositories": [
            https://github.com/donnemartin/system-design-primer.git
           https://github.com/public-apis/public-apis.git
           https://github.com/vinta/awesome-python.git
           https://github.com/TheAlgorithms/Python.git
           https://github.com/ytdl-org/youtube-dl.git
           https://github.com/tensorflow/models.git
           https://github.com/nvbn/thefuck.git
           https://github.com/pallets/flask.git
           https://github.com/django/django.git
           https://github.com/keras-team/keras.git
           https://github.com/jakubroztocil/httpie.git
           https://github.com/josephmisiti/awesome-machine-learning.git
           https://github.com/ansible/ansible.git
           https://github.com/psf/requests.git
           https://github.com/scikit-learn/scikit-learn.git
           https://github.com/minimaxir/big-list-of-naughty-strings.git
           https://github.com/scrapy/scrapy.git
           https://github.com/ageitgey/face_recognition.git
           https://github.com/soimort/you-get.git
           https://github.com/home-assistant/core.git
           https://github.com/shadowsocks/shadowsocks.git
           https://github.com/python/cpython.git
           https://github.com/521xueweihan/HelloGitHub.git
           https://github.com/deepfakes/faceswap.git
           https://github.com/XX-net/XX-Net.git
           https://github.com/huggingface/transformers.git
           https://github.com/apache/incubator-superset.git
           https://github.com/Avik-Jain/100-Days-Of-ML-Code.git
           https://github.com/testerSunshine/12306.git
           https://github.com/isocpp/CppCoreGuidelines.git
           https://github.com/0voice/interview_internal_reference.git
           https://github.com/certbot/certbot.git
           https://github.com/floodsung/Deep-Learning-Papers-Reading-Roadmap.git
           https://github.com/apachecn/AiLearning.git
           https://github.com/pandas-dev/pandas.git
           https://github.com/getsentry/sentry.git
           https://github.com/localstack/localstack.git
           https://github.com/faif/python-patterns.git
           https://github.com/google-research/bert.git
           https://github.com/facebookresearch/Detectron.git
           https://github.com/fxsjy/jieba.git
           https://github.com/3b1b/manim.git
           https://github.com/fighting41love/funNLP.git
           https://github.com/ycm-core/YouCompleteMe.git
           https://github.com/0xAX/linux-insides.git
           https://github.com/openai/gym.git
           https://github.com/satwikkansal/wtfpython.git
           https://github.com/pypa/pipenv.git
           https://github.com/littlecodersh/ItChat.git
           https://github.com/donnemartin/interactive-coding-challenges.git
      ] 
    },
    {
      "id": "reacthooks",
      "image": "TODO",
      "name": "React Hooks",
      "value": "repogroup:reacthooks"
      "description": "https://reactjs.org/docs/hooks-intro.html
            Search examples:
                 File's a hook, useState, is imported in: "
                           repogroup: reacthooks: import [^;]+useState[^;]+ from 'react'
                 Example usages for useState:
                           repogroup: reacthooks: useState({:[string]})
           ",
      "repositories": [
            https://github.com/rehooks/awesome-react-hooks
            https://github.com/testing-library/react-hooks-testing-library
            https://github.com/react-hook-form/react-hook-form
            https://github.com/facebook/react/tree/master/packages/eslint-plugin-react-hooks
            https://github.com/trojanowski/react-apollo-hooks
            https://github.com/beautifulinteractions/beautiful-react-hooks
            https://github.com/alex-cory/use-http
            https://github.com/yuchi/hooks.macro
            https://github.com/wsmd/react-use-form-state
            https://github.com/diegohaz/constate
            https://github.com/gragland/usehooks
            https://github.com/streamich/react-use
            https://github.com/kentcdodds/react-hooks
            https://github.com/taniarascia/react-hooks
            https://github.com/nikgraf/react-hooks
            https://github.com/wsmd/react-use-form-state
            https://github.com/kitze/react-hanger
            https://github.com/ilyalesik/react-fetch-hook
           https://github.com/streamich/use-media
           https://github.com/alex-cory/react-useportal
           https://github.com/csfrequency/react-firebase-hooks
           https://github.com/Andarist/use-onclickoutside/
           https://github.com/flepretre/use-redux
      ] 
    },
    {
      "id": "deeplearning",
      "image": "TODO",
      "name": "Deep Learning",
      "value": "repogroup:deeplearning"
      "description": "https://reactjs.org/docs/hooks-intro.html
            Search examples:
                TensorFlow example
                 PyTorch example: "torch.tensor(any number of args)",
                 Keras example: ""
           ",
      "repositories": [
            https://github.com/tensorflow/tensorflow
            https://github.com/pytorch/pytorch
            https://github.com/keras-team/keras
            https://github.com/opencv/opencv
            https://github.com/BVLC/caffe
           https://github.com/yunjey/pytorch-tutorial
            https://github.com/aymericdamien/TensorFlow-Examples
            https://github.com/ZuzooVn/machine-learning-for-software-engineers
            https://github.com/exacity/deeplearningbook-chinese
            https://www.techleer.com/articles/547-top-10-deep-learning-github-repositories-2018/
            https://github.com/floodsung/Deep-Learning-Papers-Reading-Roadmap
            https://github.com/terryum/awesome-deep-learning-papers
            https://github.com/Kulbear/deep-learning-coursera
            https://github.com/ChristosChristofidis/awesome-deep-learning
            https://github.com/deepfakes/faceswap
            https://github.com/Avik-Jain/100-Days-Of-ML-Code
            https://github.com/madewithml/basics
            https://github.com/ageron/handson-ml
            https://github.com/donnemartin/data-science-ipython-notebooks
            https://github.com/fastai/fastai
            https://github.com/CorentinJ/Real-Time-Voice-Cloning
            https://github.com/CMU-Perceptual-Computing-Lab/openpose 
      ] 
    },
    {
      "id": "npm",
      "image": "TODO", 
      "name": "Popular npm packages",
      "value": "repogroup:npm",
      "description": "Search over the 100 most depended-upon npm packages, pulled from [https://www.npmjs.com/browse/depended](https://www.npmjs.com/browse/depended). Search through to find usage examples and best practices from these heavily relied-on repositories.\n\nConstruct your query with regular expressions or exact matches, and narrow your scope using filters such as `file:` and `repo:`. See the [code search documentation](https://about.sourcegraph.com/docs/search) for all search features and filters. To search in this scope from elsewhere on Sourcegraph.com, include `repogroup:npm` in your query.\n\n**Example queries**:\n * Find linters and lint configurations used: [`repogroup:npm f:package.json \\\"lint\\\"`](https://sourcegraph.com/search?q=repogroup:npm+f:package.json+%5C%22lint%5C%22)\n * Find usages of `$.ajax` and `$.getJSON:` [`repogroup:npm -repo:^github.com/jquery/jquery$ \\$\\.(ajax|getJSON)`](https://sourcegraph.com/search?q=repogroup:npm+-repo:%5Egithub.com/jquery/jquery%24+%5C%24%5C.%28ajax%7CgetJSON%29)\n\n\n"
    },
    {
      "id": "angular",
      "image": "TODO", 
      "name": "Angular",
      "value": "repogroup:angular",
      "description": "Repositories by the Angular team.\n\nCompose a search query using regex or exact matches, and narrow your search using filters such as `file:` and `repo:`. Read the [code search documentation](https://about.sourcegraph.com/docs/search/) to see all search features and filters.\n\nExamples:\n\n * Search for usages of Constructor or ngOnInit: [`repogroup:angular (ngOnInit|Constructor)`](https://sourcegraph.com/search?q=repogroup:angular+%28ngOnInit%7CConstructor%29)\n\n * Find commits with messages containing \"QueryPredicate\": [`repogroup:angular message:\"QueryPredicate\" type:diff`](https://sourcegraph.com/search?q=repogroup:angular+message:%22QueryPredicate%22+type:diff)"
    },
    {
      "id": "goteam",
      "image": "TODO", 
      "name": "Go code by the Go team",
      "value": "repogroup:goteam",
      "description": "Search across high-quality Go code written by the Go team. Compose a search query using regex or exact matches, and narrow your search using filters such as `file:` and `repo:`. Read the [code search documentation](https://about.sourcegraph.com/docs/search/) to see all search features and filters.\n\nExamples:\n * Search for usages of the Retry-After header in non-vendor Go files: [`file:\\.go -file:vendor/ Retry-After`](https://sourcegraph.com/search?q=repogroup:goteam+file:%5C.go+-file:vendor/+Retry-After++)\n * Find examples of sending JSON in a HTTP POST request: [`repogroup:goteam file:\\.go http.Post json`](https://sourcegraph.com/search?q=repogroup:goteam+file:%5C.go+http.Post+json)"
    },
    {
      "id": "ethereum",
      "image": "TODO", 
      "name": "Core Ethereum code",
      "value": "repogroup:ethereum",
      "description": "Search across core ethereum code. Includes repositories from the ethereum, ethereumJS, and ewasm organizations. Compose a search query using regex or exact matches, and narrow your search using filters such as `file:` and `repo:`. Read the [code search documentation](https://about.sourcegraph.com/docs/search/) to see all search features and filters.\n\nExamples:\n * Search for usages of `CALLCODE` or `DELEGATECALL`: [`repogroup:ethereum case:yes (CALLCODE|DELEGATECALL)`](https://sourcegraph.com/search?q=repogroup:ethereum+case:yes+%28CALLCODE%7CDELEGATECALL%29)\n * Find usages of `ECDSA`: [`repogroup:ethereum ECDSA`](https://sourcegraph.com/search?q=repogroup:ethereum+ECDSA)\n * Usages of `eth_sendTransaction` or `eth\\_call`: [`repogroup:ethereum eth_(sendTransaction|call)`](https://sourcegraph.com/search?q=repogroup:ethereum+eth_%28sendTransaction%7Ccall%29)"
    },
    {
      "id": "kubernetes",
      "image": "TODO", 
      "name": "Kubernetes",
      "value": "repogroup:kubernetes",
      "description": "TODO
             Prometheus related eg?"
    }
]

Repository group lists:

rvantonder commented 4 years ago

Note ethereum group already exists, but I think it was added a long time ago, so it should probably be checked over.

Oh, also goteam, crates, and k8s for the others listed above. Should probably review the existing lists.

poojaj-tech commented 4 years ago

Sounds @rvantonder, I saw the Ethereum page.

I made a list of what already exists from what I found:

Rust crates, Popular NPM packages, Angular, Ethereum, Go
top 100 GH repos per language by stars, as of 05-2020 Rijanrd pulled with https://github.com/rvantonder/github-scripts
I see Kuberentes repos pulled with some scripts too. eg. curl https://api.github.com/orgs/kubernetes-csi/repos | jq ' .[] |= "github.com/" + .full_name'

I'm referencing these lists to curate some of items above + adding in a few example searches.

christinaforney commented 4 years ago

Moving languages data we have in the RFC here instead:

(Curated based on usage + ⬆️ recent growth from Octoverse, Stackoverflow survey):

JavaScript,
Typescript ⬆️,
Python ⬆️,
Java,
PHP,
C#,
C++,
Shell,
C,
Ruby,
Kotlin ⬆️,
Bash/Shell,
PowerShell ⬆️,
Go ⬆️,
Assembly ⬆️,
Swift,
R,
Haskell,
Lua,
HTML,
CSS

Additional languages we should add based on Top monthly languages base for file type searches over last month on Sourcegraph:

Go
Python
Java
Javascript
C
C++
Typescript
Markdown
JSON
C#

christinaforney commented 4 years ago

Repository group pages:

Repository groups content docs

poojaj-tech commented 4 years ago

@christinaforney for Excel data on top languages it's aggregated number of users per month (versus per day) for file type searches right now.

rrhyne commented 4 years ago

I added 'short_description' to the content specification to support the repository group cards required for home and repository group pages.

poojaj-tech commented 4 years ago

@rrhyne we would use a 'short_description' only on the home page right? For repository group pages the longer (2-3 lines) 'description' field would be displayed.

rrhyne commented 4 years ago

Correct.

On Wed, Jun 17, 2020 at 12:12 PM Pooja Jain notifications@github.com wrote:

@rrhyne https://github.com/rrhyne we would use a 'short_description' only on the home page right? For repository group pages the longer (2-3 lines) 'description' field would be displayed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sourcegraph/sourcegraph/issues/11539#issuecomment-645568527, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEDVBDZ35ABSQ6ELHNONKLRXEITDANCNFSM4OABJ4MQ .

poojaj-tech commented 4 years ago

Update here, I'm decoupling collecting search examples from curating repository groups so it's quick to test out examples on what exists on sourcegraph.com today.

Search examples a. Have some good examples for Python and React Hooks. b. Will add good Android egs. and some for deep learning. c. Going to post in a channel + tag people who may be able to help add more in the doc, esp for Go / Kubernetes / C / Other top languages
New Repo Group Lists a. To search over these new repo groups, we'll need to commit a diff. Will add them as experimental to site config tomorrow. b. I could download these on a local instance but it sounds cumbersome. Rijnard cautioned against downloading more than ~20-50 at a time.

daxmc99 commented 4 years ago

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.18 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

Thank you

christinaforney commented 4 years ago

Closing this since we've done a first pass and release of pages.

sourcegraph / sourcegraph-public-snapshot

MVP repository group content curation #11539