why moderate the registry?

[boneskull]

Perhaps there's some technical reason why packages must be moderated?

[ivankravets]

Yes, there is. Library submitters have often mistakes in library.json, such as: unnecessary keywords, used path to own/forked repo instead original, invalid fields which defines source location within repo.

[boneskull]

Well, to your points:

unnecessary keywords

Why does it matter?

used path to own/forked repo instead original

Why does this matter? If the original repo doesn't or won't add a library.json, must users wait until they do?

invalid fields which defines source location within repo

I don't quite understand that, but surely you can automate library.json validation.

[boneskull]

Moderating/curating a registry doesn't really scale, but I'm sure you know that.

Is there another example of a (manually) moderated/curated registry which is successful which you can site?

Is it possible to piggyback on another registry?

Is it possible to allow use of alternate registries?

[boneskull]

Specific problem: I want to use Adafruit's MAX6675 lib. It doesn't have a library.json. FWIW, I submitted a PR for it, but until anything happens on that front, I'm stuck bundling the lib manually (in my lib/ dir) with my project.

Another alternative perhaps is namespacing. That way I could use boneskull/max6675 and not interfere with whatever Adafruit is doing (or not doing).

[dave-newson]

I think the process of waiting for moderation is annoying and unnecessary. It's 2016 and there's 7 billion people on the planet. It doesn't make sense that I have to wait for the internet police to green light the availability of my opensource code. Don't you ever go on holiday?

Moderation doesn't make sense

As you've said, the the only operation you perform in moderation is to validate the library.json file, so we can assume you don't actually look at the suitability, quality, or political ramifications of the library.

With this in mind, it also doesn't make sense for you to reject a library based on the keywords it uses, as you aren't aware of the full context of the library. If a keyword really presents a problem you can PR it away from the source project like anybody else.

Moderation bait-and-switch

The moderation process is also pretty worthless after the initial submission. A repository owner can cock-up the library.json file and put whatever they like into subsequent updates. You can't possibly moderate every commit manually. @boneskull is on the money when he says the current approach won't scale.

Follow established practices

Instead, why not model your package management system on others which already exist and have been proven to work and scale, such as npm and composer (packagist.org)?

These systems only require you to login before you can submit your package, which is then automatically validated and added to the repository. No considerations are made about the code itself.

The logging on process is really only a formality to avoid automated attacks against the repos, which as far as I can tell you're already vulnerable to.

One of the only issues I can is potential for people to pollute the name pool in PlatformIO's repository. You can fix this by requiring the library.json names to be namespaced themselves, similar (or exactly the same as) the github repo name. For instance, dave-newson/derp, rather than just letting me create derp.

The repository (you) can still reserve the right to remove packages if they cause problems, break T&Cs, or generally wreck up the place. Everything else should be automated and hands-off.

@boneskull's other issue (namespaced package name, so he can use his own repo) is a gripe I've had for a while, and I'll open up a separate issue for this as it's not related to the package moderation process.

[ivankravets]

See results with disabled moderation:

• http://platformio.org/#!/lib/show/561/Json%20Streaming%20Parser
• http://platformio.org/#!/lib/show/562/ESP8266%20Oled%20Driver%20for%20SSD1306%20display
• http://platformio.org/#!/lib/show/563/ESP8266%20Weather%20Station

And this is only the one day.

[ivankravets]

I'm agree that this is my problem. Need to explain what is "Library name" and propose a few examples. Also, maybe need to disable "spaces" in name field.

What do you think?

Ping @squix78

[boneskull]

About that piggybacking idea.

Write a registry server which takes a snapshot of a library, injects a setup.py file into it, tarballs the result and uploads it to pypi. Maybe prefix the lib name or something.

registry client wraps pip with whatever custom voodoo is necessary to install a lib.

Assuming it works sounds reasonably low-maintenance, right?

Alternative: do this with npm instead.

[ivankravets]

I don't know why you compare each time PHP Composer or NPM with library.json. They are different things. Those package managers work with WELL-STRUCTURED code for the package. They have the unique rules for all developers. In PlatformIO case, we can't force people to keep "OUR OWN PROJECT STRUCTURE" which will be ready for builder without changes.

Yes, we support 2 library project layouts. See https://github.com/platformio/platformio/blob/develop/platformio/commands/init.py#L196:L209. If developer has library in this layout and put it to the ROOT of repo, then PlatformIO will build it without problems.

However, take a look at http://platformio.org/#!/lib/search?query=keyword%253A%2522i2cdevlib%2522. Here are located a lot of enormous I2C libs by @jrowberg in the single repo https://github.com/jrowberg/i2cdevlib

The philosophy of PlatformIO is to propose developers and users freedom. We try to support different boards, platforms, host OS, IDE and etc. User SHOULD HAVE a choice.

In this case, library.json proposes 2 fields (include, exclude) where developers can specify the source code location within archive, repository and etc.

Conclusion. There is no problem to remove moderation. But I can't imagine how PlatformIO Library Registry will look after it. Maybe, we will lost all our library users because we will have 10k libs, and only 200 will work properly.

P.S: We are working on the own auto-test future for the our libs. If we release it, we will disable moderation because library submitter will see information about "Does my library compile without errors"?

[boneskull]

Conclusion. There is no problem to remove moderation. But I can't imagine how PlatformIO Library Registry will look after it. Maybe, we will lost all our library users because we will have 10k libs, and only 200 will work properly.

I think this is a non-issue. What percent of packages on npmjs.com do nothing or don't work?

[dave-newson]

I don't know why you compare each time PHP Composer or NPM with library.json. They are different things. Those package managers work with WELL-STRUCTURED code for the package. They have the unique rules for all developers.

Sorry, but this isn't true. Neither of those package managers enforce rules, and indeed php is one of the most loosey-goosey languages and communities you'll ever meet.

The reason there's a semblance of structure is due to people choosing to use a particular framework or standard, not because composer makes them do it.

In fact, this has made me realise what the crux of the problem is; you're both the framework and the package manager. You have an invested (and perfectly legitimate) interest in making every library compatible with your framework. This requires your moderation, testing, and verification.

Our problem is we see this as unnecessary. We don't need you to police the package repository because we can see for ourselves if the library has examples, tests, outstanding issues, etc. Your enforced process is simply gumming up the ability for us to quickly share our code.

We also don't really care about how much is in the repository, because we have google. We use it daily to find needles in haystacks, and often we fond github repos before we find it on PlatformIO.

With all that said, you can have your cake and eat it, too. Simply add an extra flag to library.json which means "PlatformIO standards compliant".

You then feed any marked package through the verification process and rubber stamp it on the other side. This way you can add a filter to the repository which only shows verified libraries, and we can add all our untestable cruft

[ivankravets]

@dave-newson why do you need library in PlatformIO Library Registry which you can install via platformio lib install but you can't build it platformio run? I don't know which side is better for new user: to install not working library via PlatformIO or clone it to lib directory from the project and make ability to fix that structure?

It doesn't mean that I'm not agree with you totally. You are right in most cases that moderation isn't good. As I've said before:

• need to make changes to documentation with better explanation for each field
• try to build library before "skip moderation"

See https://github.com/platformio/platformio-api/issues/17

[boneskull]

Agree w @dave-newson (again). There is nothing well-structured about an npm package other than it has a package.json.

Attempting to verify packages even compile (for what boards? Cross-compiling on what architecture?) seems like a rabbit hole you really don't want to go down. Next people will be complaining they can't register their lib because your build system is broken.

Leave it up to the lib authors to use CI or not. Any npm package author worth their salt is going to do this anyway.

Whether or not a given lib works for every user of PlatformIO is irrelevant. Validate library.json, ensure it has the required fields, and leave the rest up to the community. If a lib doesn't work, expect the issue to be taken up with the lib author.

[ivankravets]

There is nothing well-structured about an npm package other than it has a package.json.

They have main field or similar to it. All logic in that scripting languages are based on "include", where Interpreter automatically build required script and go, go. In C/C++ #inculde and *.cpp are not equal. Some developers decide to use own CPPATH (Include PATH for C Preprocessor), others like to keep inc folder outside src and use global #include <>. Only developers of the lib knows where are located includes, which source file should be built and which should be skip. Sorry, it's hard to explain why "run main.js" != "run main.cpp"

Attempting to verify packages even compile (for what boards? Cross-compiling on what architecture?) seems like a rabbit hole you really don't want to go down.

Library owner specifies compatible development platforms and frameworks. We can choose the most popular boards in pair platform-framework and test it.

[dave-newson]

@ivankravets I'm on the opposite site of the coin, I don't know why you'd expect all libraries to work with platformio run. If I write a general-purpose library it might not work without the interfaces being completely fleshed out, which is the responsibility of the implementer and not the library.

Another issue would be libraries to interface proprietary code. The interface library might be MIT (3rd party) but the proprietary library isn't available to you, so you can't compile anything meaningful.

I'm with @boneskull with expecting library owners to implement CI, and not PlatformIO. That being said, I'd fully support PlatformIO having a "standards" test which can verify and mark certain libraries as "PlatformIO ready".

I'm just envisioning something basic like this:

{
    "standards": {
        "platformio-1": { "src": "/src" }
    }
}

If you change your standard you can make people start using "platformio-2" instead.

In future other developers can add other standards if they so desire; this might include stuff like code style linting, or interface hinting.

I don't mean to twist your words, but as you said:

The philosophy of PlatformIO is to propose developers and users freedom. We try to support different boards, platforms, host OS, IDE and etc. User SHOULD HAVE a choice.

The proposed change is to let any user submit any thing, without having to conform, conform conform. PlatformIO can still enforce their own framework standards by requesting library owners flag their library as platformio-1, and then PlatformIO can verify the library and wave a big green flag over it in the Registry.

Edit: Hell, give us an upvote/downvote button in the Registry too while you're at it. If a library gets downvotes then raise a "here be dragons" flag over it, and maybe alert a Registry moderator to investigate.

[boneskull]

They have main field or similar to it. All logic in that scripting languages are based on "include", where Interpreter automatically build required script and go, go. In C/C++ #inculde and *.cpp are not equal. Some developers decide to use own CPPATH (Include PATH for C Preprocessor), others like to keep inc folder outside src and use global #include <>. Only developers of the lib knows where are located includes, which source file should be built and which should be skip. Sorry, it's hard to explain why "run main.js" != "run main.cpp"

I'm not sure I really understand what you're getting at, no.

I will concede that yes, package.json is not the only structure that npm imposes; if an npm package A has dependencies of its own, those dependencies are installed in the node_modules subdir of A, which itself is in a node_modules subdir A of the application which depends upon A (this is valid for npm 2.x; npm 3.x does it differently, but the point is all dependencies of anything are always installed in a node_modules folder). Node.js looks in node_modules for dependencies.

If someone wishes to install their dependencies elsewhere, they can write a script which moves them somewhere else, but they'll also have to ensure their application can find them. That said, there is a default behavior in Node.js, which includes looking for files in node_modules. The fact that npm installs dependencies there is not an accident.

A garden-variety C/C++ program won't necessarily have such a thing. Libs are here, includes are there, you can change them, etc, etc. I understand that.

But PlatformIO libs aren't garden-variety C/C++ libs. Or at least, I don't think they should be.

If PlatformIO decided on a standard place to install dependencies--local to a project--you wouldn't worry about if Joe Developer wants to put his includes somewhere else. Why? Because if Joe Developer wants to put stuff somewhere else, his lib will not be usable out-of-the box, and people will complain about it until he fixes it, or forks his library. PlatformIO will expect all of the dependencies to be in a certain place, or set of places, and if they aren't present, it will fail to compile/upload.

I feel like trying to support a bunch of IDEs and the various places which they want to put files will also not scale. You're putting the burden on yourself to do it, instead of saying "Hey everyone, look at PlatformIO! It's a really good idea, isn't it? Ask the developer of your favorite IDE to add native PlatformIO support! In the meantime, here are some workarounds!" Now, you wouldn't have to worry about wherever IDEs want to put files, because that's their problem now.

If necessary, you can develop plugins for various IDEs; but don't design PlatformIO around IDEs!

PlatformIO is a great idea. What I don't want to be happen is you guys end up bogged down by mundane, unnecessary tasks like policing a registry. The ecosystem will police itself. Give us cool stuff instead.

[boneskull]

Imagine a future wherein people dump Arduino IDE in droves because they refuse to support libs structured for PlatformIO. :palm_tree:

[ivankravets]

Guys, which problems do you have with the current moderation policy? I moderate all pending libs within 1 hours. What is more, we don't have a lot of new libs.

[dave-newson]

I think what we're asking for is:

1. Remove the human element of library moderation

No matter how good of a job you're doing, it won't scale, you need to sleep, etc, etc. You've already said you're working towards this with automated validation, so that's great.

2. Allow more freedom in the library submissions

This has only been raised because it was the argument for why there are strict moderation practices. We'd like you to allow libraries in the Registry which don't specifically conform to PlatformIO's standards.

3. Highlight/mark libraries which DO follows PlatformIOs standards and pass tests

This is a great way to allow item 2 without losing PlatformIOs inter-compatability.

For libraries which state they are "PlatformIO Ready" in the library.json, run your automated validation/compile tests, and mark the library as being "PlatformIO Ready" in the Registry.

This gives you that warm fuzzy feeling that newbies know to stick to the PlatformIO Ready libraries and they will just work, while it also allows others to register libraries which don't follow the prescribed standards.

In addition, if you change your framework's standard later in life, you can still correctly verify if libraries follow the PlatformIO1 or PlatformIO2 standards and label them appropriately.

[JerrySievert]

also, how are libraries updated in platformio library registry? i noticed that it does not occur with a git push.

is there a mechanism to trigger the update in the library registry?

[boneskull]

I am in agreement with @dave-newson, except #3 seems like unnecessary overhead. A package may or may not be "compliant" at any given commit or version. Furthermore as PlatformIO changes, older lib versions which were once compliant may no longer be.

Again I feel the community will work it out on its own. If a lib fails to compile, well, that's what GitHub issues are for.

[boneskull]

It's worth mentioning again that any stamp of approval a lib may get won't guarantee that it actually works with whatever weird edge case or platform people are trying to use it on. As such such a thing loses value.

[JerrySievert]

as an aside, i agree with @boneskull and @dave-newson.

[ivankravets]

Friends, OK, I'll remove moderation later after improving library.json validator for http://docs.platformio.org/en/latest/userguide/lib/cmd_register.html

P.S: We don't have a lot of new libraries. I want to focus on PlatformIO IDE and PlatformIO 3.0.

P.S.S: Have you tried PlatformIO IDE for Atom (beta)?

[JerrySievert]

@ivankravets awesome! thank you! just curious how often libraries get updated on platformio, and how to force an update if a new version is released.

also, i will definitely look at the IDE for Atom, but will note that at the moment i'm using the clang plugins and the code intelligence is working very nicely (once you set up .clang_complete).

[thegoldenmule]

It's been a long time... Any update on removing moderation? I've been waiting for "ages" for my libraries to get through.

[ivankravets]

@thegoldenmule could you point to a library in PlatformIO Registry?

[thegoldenmule]

I'm not sure what you're asking. How can I help?

[ivankravets]

@thegoldenmule Please share the links to your libraries in http://platformio.org/lib

[thegoldenmule]

These were the libraries you put through just this morning (http://platformio.org/lib/show/1971/DataFlash and related). Thank you by the way. As has been discussed thoroughly above, most package systems don't require human intervention, meaning you can publish and consume within seconds. I put "ages" in quotes because, while I waited no more than a day or two for the packages to be published, I had to wait at all.

[boneskull]

@ivankravets Thanks for this!

[ivankravets]

The new PlatformIO Registry has been deployed and there is no more moderation. See updated docs https://docs.platformio.org/en/latest/librarymanager/creating.html#publishing