Some questions and suggestions about the workflow

oovm commented 6 years ago

1.How can I start a mirror server?

I want to build a mirror server in China with localization and download acceleration.

What do i need? I already have the server and the domain.

2. How about use github issue instead of review queue

The workflow used is a bit complicated, use github issue instead of fork may make things easier.

This can also avoid merge conflicts.

Authors just provide a build script and a markdown application in issue.

Then the maintainers finish the review, run the build script, finally, upload to the git.

If you think it's OK, I'll have a try.

3. Is it possible to have a Continuous Integration Service

Continuous Integration means when I do a git push, the CI server will run build script and upload the built package.

In this way we can synchronize many packages from different git repo.

4.Why not use git LFS or submodule for `*.paclet`?

b3m2a1 commented 6 years ago

The only real issue you'll have is that all the pages point to https://paclets.github.io/PacletServer as their base URL. I can cook in a nice way in the build script to build relative to other URLs too, though. It should be as simple as a 1 line change. My hope is to have an interface paclet that could support this kind of thing too. If this sounds good submit an issue for it so that I can keep track of what's happening and I'll write it up.
I'd love to see an example of what you're thinking of. I'm open to any and all ways to make people not have to fork the repo. If there's a workflow that operates via the GitHub API I can provide immediate integration, too. The main thing I want to be sure of is that a) I don't have to execute arbitrary code that other people write b) there's a way to hold a reference to whoever committed the paclet that I can extract and log for security reasons. Give me an example of your proposed workflow and I'll see what I can do to make it happen.
This will be harder as I build the site via Mathematica itself from the .paclet files. I don't really want to try to run something via the cloud... I don't really know how I'd handle this.
This one was actually on my radar. I've been looking into ways to shrink the repo size beyond simply shallow cloning. Submit a standalone issue and I'll comment there once it's done.

b3m2a1 commented 6 years ago

@GalAster forgot to ping you. Do you have something for 2? I'm hoping to have a good block of time to work on this sometime ~2 weeks from now and would like to use the intervening time to think about it.

oovm commented 6 years ago

Sorry for late, things are not as simple as I think, lol...

Create a new super repo for managing all repos on github.

Authors just submit their applications on that repo, then the maintainers will finish the review and add to the scheduled tasks' queue.

Repo1Name
  |-- Source(git submodule)
  |-- PacletBuilder.sh/.wls
  |-- BuildConfig.m/.json/.yml
  |-- Readme.*
  ...
Repo2Name
.gitmodules
...

And I've think some possible plans before, but now I change my mind, the following code just for reference.

PacletServerAutoBuilder::ussage="Add scheduled build tasks for servers.";
PacletServerUpload::ussage="Upload the created paclets.";
Begin["AutoBuilder`"];
PacletServerAutoBuilder[ass_Association]:=Switch[
    ass["plan"],
    "DownloadOnce",ass["para"]//DownloadOnce,
    "GithubRelease",ass["para"]//GithubRelease,
    "GithubReleaseHook",ass["para"]//GithubReleaseHook,
    "GithubRepo",ass["para"]//GithubRepo,
    "GithubRepoHook",ass["para"]//GithubRepoHook,
    "SomethingElse",ass["para"]//SomeMethod,
    _,"Custom"
];

DownloadOnce::ussage="Build paclets from download link.";
DownloadOnceExample=<|
    "plan"->"DownloadOnce",
    "para"-><|"url"->"example.com"|>
|>;
DownloadOnce[ass_Association]:=Module[
    {},
    URLDownloadSubmit[
        ass["url"],CreateFile[],
        HandlerFunctions-><|"TaskFinished"->PacletServerUpload|>
    ]
];
End[];

oovm commented 6 years ago

The problem lies in the builder part. The variety of projects is too complicated.

What if the author forgets to change the version number?

What if the author's project is updated then broken?

What if the author's project can not be built at all? Just a single file script, should we auto add the construction file?

So I think what I thought before was too simple.

oovm commented 6 years ago

Some parameters

<|
    "plan"->"",
    "para"-><|

    |>
|>

url: url where to get the source
verfix: Ture/False auto fix version number, a.b.x_ -> a.b.x+1
pacletfix Ture/False build forcibly with default template if True
rate: time(s) detection update frequency if no hook
release True/False add a release to source repo, it's unsafe if use github secret key, but we can add a co-author bot.
author: auto/custom can be got from PacletInfo
...

Some API

https://developer.github.com/v3/repos/commits/ https://developer.github.com/v3/repos/releases/

b3m2a1 commented 6 years ago

I will see what I can do, assuming the project lives originally as a git repo this is possible, but rather complex. I have implemented workflows for automatically converting a GitHub repo to a paclet before. I can try to repurpose that here.

b3m2a1 commented 6 years ago

@GalAster I’ve added partial support for this. The build process will automatically download and build a paclet one it gets added to a config file. Then people can simply submit an issue specifying where this should come from.

oovm commented 6 years ago

Can I join paclets, I'll divide these into small issues.

paclets / PacletServer