MarlinDev as a separate project?

[thinkyhead]

@Wackerbarth, @fmalpartida, and I have been discussing how best to provide separation between development, code integration, release candidates, and releases. Among the ideas being floated is to create a separate project (MarlinFirmware/MarlinDev) where all development would be done, while keeping Integration and Release branches in the current repository. Richard can articulate better than me the added benefits of this approach. One of the main ones is separation of issues and PRs.

In any case, it would require @MarlinFirmware to move all the active @MarlinFirmware/collaborators, @MarlinFirmware/owners, and @MarlinFirmware/testers over to the new repo.

Let's discuss this concept.

[avluis]

This is actually a very to thing to consider - just the separate issue tracking for the Release branch and Dev branch is enough reason to do this.

[boelle]

no need to move people... the different teams are all top level

[boelle]

repro created. access should be the same. let me know if not

[boelle]

could any of the branches here go? just so there is less messing and confusing here

[Wackerbarth]

@boelle -- Would you please change the default branch for MarlinDev to "master"?

[boelle]

yep first thing in the morning

just gone to bed

Den søndag den 2. august 2015 skrev Richard Wackerbarth < notifications@github.com>:

@boelle https://github.com/boelle -- Would you please change the default branch for MarlinDev to "master"?

— Reply to this email directly or view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/2578#issuecomment-126959269 .

[thawkins]

I don't have any issues with this as long as somebody is going to take the responsibility on of moving distinct snapshots or release integration branches from the dev repo to the release repo. (release engineering role)

are we also planning to provide a library of downloadable zip files for key milestone tags in the releases repo, so users can avoid having to setup and install git to get the code for a release.

I would also like to raise at this point that while we are looking at methods to simplify things for endusers, it may be worth considering the following.

1) conduct a survey of users (method yet undetermined) to work out what the most common configurations are and support prebuild .hex files for these. 2) move some of the customization to runtime commands stored in the eeprom with m500 etc.

Printrbot does this successfully, ther have a universal .hex file for all their models. with the ability to set bed size and extruder offsets for autoleveling. They precompile for two extruders even on a single extruder machine. I would expect that we can come up 4-6 configs that given 2) above could hit at least 75% of the user base. LCD displays are probably the most problematic aspect. People that don't fall into those prebuilt configs would have to rebuild as they do at the moment.

On Sun, Aug 2, 2015 at 5:39 AM Scott Lahteine notifications@github.com wrote:

@Wackerbarth https://github.com/Wackerbarth, @fmalpartida https://github.com/fmalpartida, and I have been discussing how best to provide separation between development, release candidates, and releases. Among the ideas being floated is to create a separate project ( MarlinFirmware/MarlinDev) where all development would be done, while keeping Integration and Release branches in the current repository. Richard can articulate better than me the added benefits of this approach.

In any case, it would require @MarlinFirmware https://github.com/MarlinFirmware to move all the active @MarlinFirmware/collaborators https://github.com/orgs/MarlinFirmware/teams/collaborators, @MarlinFirmware/owners https://github.com/orgs/MarlinFirmware/teams/owners, and @MarlinFirmware/testers https://github.com/orgs/MarlinFirmware/teams/testers over to the new repo.

Let's discuss this concept.

— Reply to this email directly or view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/2578.

[boelle]

"@boelle -- Would you please change the default branch for MarlinDev to "master"?"

done

[boelle]

again what branches should we keep here? just so we have less clutter to confuse people and not doubles etc

[thawkins]

Branches will be driven by need, if we have new features that will take time to sort out then they will get a branch.

The only two we probaly need at the moment are master and a integration branch, when a candidate tests out in the integration branch, it would be merged into master on the release repo and then tagged.

On Sun, Aug 2, 2015, 14:49 Bo Herrmannsen notifications@github.com wrote:

again what branches should we keep here? just so we have less clutter to confuse people and not doubles etc

— Reply to this email directly or view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/2578#issuecomment-126995263 .

[boelle]

ok let me ask more clear:

what do we need in each of the repro's ?

sure some of what is in Marlin no longer needs to stay as its now in Marlindev

[Wackerbarth]

On Aug 2, 2015, at 2:38 AM, Bo Herrmannsen notifications@github.com wrote:

what do we need in each of the repro's ?

sure some of what is in Marlin no longer needs to stay as its now in Marlindev

At the present time, Marlin (is moving toward) has only 2 branches.

The “Release” branch serves as a “front page". But rather than being a “branch”, it acts more like a variable tag. At all times, it will point to the latest tagged release. (Perhaps plus one commit involving the project’s README.md)

The actual branch which has a history of all of the releases in a series has a name of the form "..x" (eg "1.0.x”)

After we reach the “1.1.0" release, “Release” will point to that. There will also be a “1.1.x” branch. It will point to the 1.1.0 tag PLUS any approved patches. And the “1.0.x” branch will still be available.

At the present time, “Development” still remains in this repository ONLY TEMPORARILY while we clean up, migrate, or otherwise dispose of the existing Pull Requests.

Because of the way GitHub handles Pull Requests and merges, we may also create temporary “Inbox” branches against which patches may be submitted. But the important thing is that only the “Release Engineering Department” will make changes to the “Release”, “1.0.x”, “1.1.x”, etc. branches. And ANY OTHER branches are “temporary” and “transient”. They are subject to deletion at any time.

On MarlinDev, the same philosophy applies. “master” is the current “front page” and “inbox” against which submissions will be made. “dev” points to the current development and the “1.1.x”-type branches will be created as appropriate.

NOTE: Both of these repositories are actually just separate views of one underlying “master” repository. Their contents are both parts of a common git tree and it is trivial to move material from one to the other using simple git push commands from your local copy of the master copy. Each of these GitHub projects appear as a “remote” to your local system.

When, hopefully very soon now, we are ready to pre-release “1.1.x” for testing, we will expose it under the “RC” branch on the “Marlin” project. That way, feedback on testing, etc. can be kept separate from development taking place on “MarlinDev”.

[boelle]

so at the moment i should not delete any branches?

[Wackerbarth]

Not on GitHub. I've already taken care of that aspect.

[boelle]

:-D

me just a noob here

btw. behind the scenes i have encouraged @Roxy-3DPrintBoard (aka Roaxana) to give a go at cleaning it up. She have done a lot of work but might need help on how to make a PR the right way

[Roxy-3D]

btw. behind the scenes i have encouraged @Roxy-3DPrintBoard (aka Roaxana) to give a go at cleaning it up. She have done a lot of work but might need help on how to make a PR the right way

Specifically... I was going to try to clean up the G28 & G29 code. There is a lot of stuff to clean up, but that is probably the most outrageous and it is an area that interests me. Perhaps we should start a separate thread to get people's comments (and help) on that topic. I guess I'll do that over in the MarlinDev area.

[boelle]

Good idea

One easy but maybe cumbersome way to do a PR is to fork, make changes... submit PR....

PR can be submitted from the web interface and also the PC based clients...

[boelle]

btw... i do only watch a very limited number of issues or my inbox get hammered full

so one way to get my attention is to do @ in front of my username... unsubscribing from this one also

[thinkyhead]

The one thing I can think of that makes this a little trickier for contributors is that all the current forks of Marlin are based on this repo and not the new one. So the new repo adds an extra layer of complexity for users who want to submit customizations from their existing forks. And, we will have to tell everyone who submits a PR now that they must create a fork from the new repo and submit their PRs to the new repo.

Should we follow up on the several PRs that exist in this repo, merge them here and then bring them over to the new repo. Or, should we post a message on all PRs that they should redo them with the new repo?

[Wackerbarth]

Yes, it is a slight complexity for the small number of individuals who are working with "before pre-release"

However, I anticipate that we will do "Testing" here in this copy and merge those resulting changes here and copy them "internally". But those details will need to be determined as we gain some experience with the setup.

As to open PR's. Let's close (with a message to resubmit on MarlinDev) any that are old. And let's merge (here) any that you find ready, or with only minimal adjustment. Then we can re-evaluate the ones that remain in that gray area between the two extremes.

[boelle]

closing this one as it has been done

[AnHardt]

Reopened because this explains a bit what is going on now. (At least for a week or two)

[a4jp-com]

Even if this topic is closed, people can still write here can't they?

[Roxy-3D]

I would appreciate a short discussion on what we did here. If you have strong feelings or good insight, please do not hold back.

Specifically, I'm a little bit concerned we made extra work for our contributors by doing this. And in fact I think it is possible this split caused one of our repositories to get messed up because of the extra complexity it added to the environment. (Big Thanks to Wackerbarth for cleaning up the mess!!!!)

Are we getting the benefits we anticipated? Is the separation between Marlin and MarlinDev really giving us benefit? Especially once we promote a Release Candidate to 'Stable' status?

[Wackerbarth]

@Roxy-3DPrintBoard -- We did this for two reasons. 1) We wanted to try to separate "support" issues on Released firmware from "development" issues. 2) There are significant differences in file structures, build instructions, etc. It was felt that some separation would help keep individuals from attempting to apply the wrong set of instructions.

And remember that we were not supposed to have this RC period drag out "forever". The intention was that Marlin would become an almost static "site". The "extra work" related to having any stability in releases and still allowing active development will always exist. And, if you get away from using the github pull requests for the back porting, it really isn't very bad.

IMHO, misusing the github workflow, whether in one repository, or many, is the source of much of the hassle.

[Roxy-3D]

OK. Two questions for @Wackerbarth :

1) When the RC period closes and we promote something to a 'Stable' status, does the reason for the split go away in your mind?

2) Is the split really helping to separate support issues from development issues? I'm OK with the way it is right now, but there is a lot of chatter on each side that really belongs on the other side. What we have done is made it so the contributors here need to check and be active in two areas instead of one.

[Wackerbarth]

1) No, not at all.. In fact, it becomes even more important. The instructions for building are different, features and settings will be different, etc. We will have most less-knowledgable users trying to rebuild their firmware. So the separation is really more important at that point. Additionally, the "I can't get .... to work" issues remain separated from the actual submissions which add to the code base.

2) I don't think that we are realizing many of the benefits of the split yet. There is still too much "development" being attempted in the Marlin repository.

[AnHardt]

Two Repositories.

In the order of convenience i use the GitHub-Web-interface, the Github-desktop-program, git gui, gitk and the command-line git. My normally save way to handle a GitHub repository is to fork, for example MarlinFirmware/Marlin to AnHardt/Marlin, clone AnHard/Marlin to my computer. To distinguish between the 3 levels i can use 'origin' - for AnHard, 'upstream' for MarlinFirmware and nothing for local. Now for MarlinFirmware/MarlinDev -> AnHard/MarlinDev -> ???. Use a second local folder? Then i can't transport commits from one to the other repository. Or make AnHard/MarlinDev a second source in the first folder? Then i can't use 'origin' and 'upstream' because it is not clear what is mend - Marlin or MarlinDev. Suddenly i have to deal with 4 similar looking web-addresses to type or paste on the command line and a confused GitHub-desktop-program. It's proved I do make errors then. So the current state for me is to have 3 local repository. One for Marlin, one for MarlinDev and one to let them interact. In total that's dealing with 7 repository instead of 3. A lot of work to keep them in sync. Constantly you have to answer the question - Where am I.

I know, searching for something is a forgotten art. Just to ask around and waiting for someone else to do the work is a common attitude. In some cases the one who answers the question is knowing the fact by hart - tat's luck. In the other case you have someone actively searing in the available sources. A experienced supporter has usually an idea how to narrow down the results, (author issue, PR, timespan, keywords, ...) With two repository the supporter additionally has to remember in witch one - or has to do the search twice.

References. Instead of just typing #2797 you now paste "https://github.com/MarlinFirmware/Marlin/issues/2797" if in the other repository.

Mailfilter -> doubled

...

It piles up

Any tips?

[thinkyhead]

As this is now a foregone conclusion, I think we can close this initial topic. It may be worth revisiting in the future. For my part, I've given in (!) and I'm finding it more convenient to have the "Marlin" repo only dealing with releases, while the "MarlinDev" repo only deals with development towards 1.2.x. The main inconvenience is of course keeping the two source-trees in sync and having to make 2 PRs for every bug-fix, and keeping issues in the right places. Fortunately, so far the sources have not diverged significantly, so we can still compare the sources and catch any patches we missed earlier. Some duplication of effort will probably always be unavoidable, as in any project there are going to be bugs discovered in the next version which you'll want to apply to a point-release on the previous version.

[Wackerbarth]

It wouldn't be such an issue if the two of you would stop trying to add improvements in the 1.1 branch.

We certainly know that there will always be room for improvement. But we really need to simply "draw the line" and fix ONLY the most egregious problems in the 1.1 branch.

[thinkyhead]

It wouldn't be such an issue if the two of you would…

Geez, you try to be conciliatory and you get fingers pointed in your face.

The release will take as long as it takes. I think we all understand that. While there are still some bed-leveling and other "egregious" issues extant, it's no skin off our nose in the meantime to include some of the little improvements being made along the way. I may bitch too much about it, but the main bottleneck is not the extra effort of making two PRs for every patch (which we would have to do for 1.1.1, 1.1.2, etc. anyway). The main bottleneck is that everyone is doing this in their spare time, so we do a lot of waiting for testers and patches.

Anyway, I did almost zero on this for a couple months. Now I'm spending a couple hours a day. This will get things moving a lot faster, even if there are a few extra PRs involved. So count yer blessings!

[a4jp-com]

I think the message from @Wackerbarth was a bit of a joke. You're doing a great job and any help is greatly appreciated. Do you need any help testing anything?

[Roxy-3D]

@thinkyhead Everything is good... Please keep help fixing bugs!

[a4jp-com]

Just out of interest, how is the bug count going now?

[thawkins]

I would like to ask if somebody could do a quick write up on a wiki page of the process from fix to PR etc, i tried to do it once and got confused and brow beaten about it, so i decided not to bother again, i suspect you folks would get a lot more help if the onboarding process for devs was easier to discover. On Feb 11, 2016 12:37 PM, "Scott Lahteine" notifications@github.com wrote:

It wouldn't be such an issue if the two of you would…

Geez, you try to be conciliatory and you get fingers pointed in your face.

The release will take as long as it takes. I think we all understand that. While there are still some bed-leveling and other "egregious" issues extant, it's no skin off our nose in the meantime to include some of the little improvements being made along the way. I may bitch too much about it, but the main bottleneck is not the extra effort of making two PRs for every patch (which we would have to do for 1.1.1, 1.1.2, etc. anyway). The main bottleneck is that everyone is doing this in their spare time, so we do a lot of waiting for testers and patches.

Anyway, I did almost zero on this for a couple months. Now I'm spending a couple hours a day. This will get things moving a lot faster, even if there are a few extra PRs involved.

— Reply to this email directly or view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/2578#issuecomment-182700778 .

[a4jp-com]

What is the address of the Wiki page you made?

[AnHardt]

@Wackerbarth It wouldn't be such an issue if the one of you would not had not stopped MarlinDev to compile in a 'normal' way.

[a4jp-com]

The version I have compiles properly. Which version is the version that doesn't compile @AnHardt? Would you like a link to my version?

[Roxy-3D]

It wouldn't be such an issue if the one of you would had not stopped MarlinDev to compile in a 'normal' way.

So... Just for the record.... I'm not doing this.... I am developing with the new file organization in MarlinDev, but I'm not using this feature.

If you want.... You can set the #defines so it uses your old Configuration.h file. You don't have to do the inheritance thing.

On Fri, Feb 12, 2016 at 7:15 PM, a4jp-com notifications@github.com wrote:

The version I have compiles properly. Which version is the version that doesn't compile @AnHardt https://github.com/AnHardt? Would you like a link to my version?

— Reply to this email directly or view it on GitHub https://github.com/MarlinFirmware/Marlin/issues/2578#issuecomment-183552058 .

[Wackerbarth]

@thinkyhead - "The release will take as long as it takes. I think we all understand that. While there are still some bed-leveling and other "egregious" issues extant, it's no skin off our nose in the meantime to include some of the little improvements being made along the way."

I cannot agree. It is "skin off our nose" when you introduce these "little improvements" because that creates an ever moving target. Unfortunately, even though the "improvements" are each well intentioned, there are many subtle interactions which are never fully vetted before they get introduced into the "product". That, IMHO, is why we are having the problems with bed leveling at this time. If, during the correction to these problems, you continue to introduce additional "improvements", the chances of having some additional problem introduced keep increasing. And, without a stable target, all of the testing that is done remains incomplete. As such, you will NEVER reach a stable, well tested, version that is worthy of being called a Release.

Therefore, if you want to introduce improvements, you really should limit them to the code base which will be a part of a future release cycle.

If, in your opinion, the current release candidate is so flawed that you feel that additional improvements can be added before the major problems are addressed, then we should consider abandoning the current release candidate and jump directly to the next cycle. But doing that implies that we include changes which come from contributors other than yourself. That code base already exists in MarlinDev. Were it not for my reluctance to introduce some significant changes to a release cycle in progress, we would have already made the initial jump to the transitional version.

You should also note that we have actually made two major changes from the code base structure of RC3. I made a deliberate effort to provide a transitional path which introduces them in stages so that it would reduce the incremental complexity for a user to migrate over.

[thinkyhead]

At the moment the number of actual BUGS being reported for RC is quite low. I think we're getting very close to release. RC4 might just be the last RC.

[Roxy-3D]

That would be nice if it turns out to be true! The Unused End-Stop / Undeployed Z-Probe problems account for about half of the problems I've seen posted. And AnHardt has some Pull Requests for that set of problems that have been very thoroughly vetted. We mostly just need to get them joined in with the rest of the bug fixes and make sure everything works as expected.

And the other issue that is very serious is the flaws in the Serial Communications where characters can get dropped. That is very serious from a user's perspective because it can stop a print at a random time with no explanation. From what I now understand about the problem, I think it is critical we get those Pull Requests into the RCBugFix also.

If we can all agree those things should be done... I'll load up RCBugFix as soon as its ready and start beating on it.

[thinkyhead]

flaws in the Serial Communications where characters can get dropped

Can that actually be prevented or are we stuck just mitigating it through the use of a checksum, OK responses, etc.?

[AnHardt]

Na. All the lost characters are detected if the line has a checksum. And is requested to resend. Normally the worst thing is a momentary stop of the head.

[AnHardt]

Yes. The overflows are away when sending the correct number of ok's. But there is another error where 'strange' strings, never sent by the host, appear in the input stream. I have no better explanation for them than a race condition between the normal reading program and the two writing interrupts. And they vanish with my patches.

[thinkyhead]

they vanish with my patches.

That's what we want! In the long run, I'm sure a flowchart will be a great help. Then we can authoritatively say "this happens at this time," and "that happens at that time." On the other hand, looking at other firmware, like Repetier, can also be instructive!

[Roxy-3D]

Na. All the lost characters are detected if the line has a checksum. And is requested to resend. Normally the worst thing is a momentary stop of the head.

I'm just nit-picking here. The 8 bit checksum is very weak. If we lose two characters it is possible for them to cancel out each other's effect on the checksum. Specifically, for example, suppose we lose two 0's in a number. The checksum would calculate to be correct. There isn't much we can do about it.

But there is another error where 'strange' strings, never sent by the host, appear in the input stream. I have no better explanation for them than a race condition between the normal reading program and the two writing interrupts. And they vanish with my patches.

I believe this 'race condition' was real. It hardly ever happened, but without the patches it could happen at any time. And the strange data (left over from previous commands) is exactly what would happen.

Then we can authoritatively say "this happens at this time," and "that happens at that time."

It is difficult to make happen, but getting an interrupt in the middle of updating those Rx and Tx buffer indexes definitely could cause problems because other code (the interrupt handler) would be using an incorrect value for the Rx and Tx buffer indexes.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.