[Discussion] Arch / Manjaro update & package testing policy

[keybreak]

Problem

• Linux Deepin release in relatively long cycles by mlestones, which are reasonably stable & tested
• Arch Deepin release very untested packages, which coredumps, break what was working, bugged etc
• Manjaro Deepin have longer cycles, but still picks up all Arch Deepin bugs, then we need to roll-back, test, try to fix bugs ourselfs etc

Followed by problems:

• @oberon-manjaro Manjaro Deepin can't release new .iso for months, which leads to problems / time wasted / frustration.
• @antechdesigns can't release new .iso because Deepin's Arch packages are breaking for months.

We need better testing before pushing Arch packages.

That is pretty obvious.

What's pushed on Arch now is very poorly tested (if at all). For example last updates caused constant coredumps and crashes of dock / control-center.

There are a lot of known bugs coming from dde-daemon, constantly breaking what was working before etc.

Most of new bugs which come from packages pushed on Arch can be avoided, if you just boot system / open updated program / check journal for errors.

---

@BLumia @felixonmars Don't get me wrong, i personally highly appreciate & respect your work, and have faith in Deepin's future. Also understand all the difficulties you're facing now while moving to deepin-kwin and resolving long-term serious issues...

But for now at least Manjaro Deepin and Arch Deepin projects are on stop, because of this situation.

We seriously need to think on how to improve testing / bug fixes for Arch packages BEFORE releasing them to public, otherwise it's just unmaintainable mess which will only grow in size with time.

[antechdesigns]

@keybreak

Totally agree, Regarding kwin just don't release any DE updates until its been implemented and if you need testers, I am sure there is enough people here that would be interested in testing.

Not a good idea to be releasing buggy updates on to peoples production machines, having said that, I have stopped reporting bugs, seems a waste of time while kwin is being pushed out.

Like @keybreak says you guys are doing great work and I am 100% confident in your team to release a great linux desktop experience.

Please don't make me go back to Winblows ;)

Good Luck Guys :)

[felixonmars]

@keybreak As I said already in a previous comment, I always test basic functionalities myself and did not encounter any coredumps here about dock or control center. If it was in any case so easy to spot it won't happen in the first place. If you have a problem like this please open a bug report with details.

I have pain in my fingers since last week so I can't really add a lengthy reply. As I already said before, the current DDE packages in Arch are treated the same way as the others (KDE, GNOME, or any other general packages), which we follow upstream releases if it's not severely broken (does't open, main function doesn't work, etc) and only try to test/fix some major issues ourselves. The others should be reported and fixed in upstream respectively. New versions always introduce other bugs and there is no point to duplicate the work onto a cutting-edge rolling distribution.

[keybreak]

@felixonmars

If it was in any case so easy to spot it won't happen in the first place. If you have a problem like this please open a bug report with details.

Nobody argues that as general rule of thumb, and as you know i personally constantly reporting bugs after almost each update, for already 3 months.

I always test basic functionalities myself and did not encounter any coredumps here about dock or control center.

New versions always introduce other bugs and there is no point to duplicate the work onto a cutting-edge rolling distribution.

Sure. there will always be bugs and some are really hard to catch as they're random or hardware specific. I'm talking about super-obvious ones, like crashing, 100% reproducible on any machine, detectable by eye of any end user.

Facts are undeniable here, clearly now something is not right, since:

1. I personally as end user & tester can very clearly see 100% reproducible bugs (which are clearly avoidable if what you're saying is the case) for each Arch / Manjaro updates release, since at least late Janary.
2. I'm not alone here, both @oberon-manjaro & @antechdesigns which have bugs reports ethics checked and very interested in Deepin - can't really keep up with downgrading faulty packages or reporting that many 0-day package release bugs (by most part since they're expected to be known / fixed immideately).
3. I'm sure @oberon-manjaro as one of Manjaro maintainers can confirm, that no other Manjaro editions (KDE / XFCE / Gnome etc) have this kind & amount of problems, or have them very rarely.

So i say we need to do something with it, to improve Deepin testing process.

I have pain in my fingers since last week so I can't really add a lengthy reply.

Sorry to hear that, take some time to fix your health! No one is terminator, and you do extreme amount of work on Arch yourself... That can lead to perception bias / errors unless you take care of yourself.

[hualet]

How about a more direct way, like IMs etc, to communicate about serious bugs? so we can release bugfixes more rapidly?

[keybreak]

@hualet At least for internal & testing team it's a really good idea to have something like telegram chat for rapid serious problems discussion / resolve :smiley:

When problem is clearly defined, faster communication would certainly help.

[keybreak]

@hualet Just few examples of testing problems (minor, but very illustrative there was far more hardcore issues with dock / settings / daemon which follow same principle):

1. Auto-merge poorly tested before public release https://github.com/linuxdeepin/developer-center/issues/1082

   If it was tested for just 2-5 minutes on actual production system, before release - this would be obvious and could be fixed immediately (or just hold this feature until it fully resolved).

2. While this issue https://github.com/linuxdeepin/developer-center/issues/831, of disappearing dock program was resolved, because it was poorly tested we still have this obvious consequence https://github.com/linuxdeepin/developer-center/issues/1009

   Could it be better tested? Absolutely, it only requires to actually test this issue - pin any program to dock and update this program 2-5 minutes of test.

---

As you can see, just little more testing prior release could resolve a lot of new issues, which otherwise wouldn't be fixed for days / weeks / months or even buried forever under more serious bugs.

We could save time to both developers and bug-reporters and certainly headache of users.

[BLumia]

Current Deepin development and release cycle (for project which under active development):

(1) Feature development and bug fixes -> (2) Code freeze (will got a new tag for QA team testing) -> (3) Fix bug we found while QA peroid, final check (will also got a new tag for release) -> (4) Release

So in a single release cycle, there will be at least 2 tags available, Arch will create a new package everytime when we got a new tag, so for a single release cycle, Arch DDE user will receive at least two update for a single package.

The tag release in (2) may or may not treat as a unstable release (I'm not sure), since developer will always test the function of a module (unit test) before we do the tag release, so normally there won't cause critical bug at most of the time. but we cannot always grantee everything is bugfree (I feel really sorry about #1082 😭, really sorry about that), so we need do QA then.

For the generic bugs, I think the better way to do is introduce better automation test to our project (not all project got automation test and the test cover is also not enough), but this is for generic issues, not a Arch/Manjaro specific issue. The Arch/Manjaro specific problem is, both our development team and QA team are working on Deepin, so there will be Arch specific problems we cannot test and fixed at the very first time.

Since Arch treat DDE the vary same as KDE/GNOME packages, I don't think the Arch package release policy should be changed, if manjaro want a better tested version for packaging, probably you may need only do packaging after Deepin got update (which means a single release cycle is done)?

[felixonmars]

Currently Arch specific changes are handled by myself right on each update. I add fix or workaround for some critical compatibility issues myself, for example in yesterday's tags https://github.com/linuxdeepin/qt5dxcb-plugin/pull/13 and https://github.com/linuxdeepin/developer-center/issues/1103 were filed.

However this can hardly be Arch/Manjaro specific. Other ports like Debian/Fedora/openSUSE will get similar issues and perhaps use my patches or wait for next release after it's merged.

Other issues are mostly unrelated to distributions and should be fixed upstream instead. Manjaro may consider what @BLumia suggests for better tested tags only.

[keybreak]

@BLumia So areas where we can improve Deepin's process:

1. Unit tests

When you refer to unit-tests those are done on per-module basis only by developer of introduced change?

Therefore i assume, that maybe unit tests should also be done again on whole tagged release (because context matters).

2. Developer's manual test

I also think that outside of QA team this is must, just at least for 5 minutes launch program / related packages in context of whole OS, check stdout / stderr / sudo journalctl -p3 -xb / Manually check core functions are actually working.

In my experience this is super-important, besides developer could also point QA team to grey-areas of testing focus (related to particular changes).

3. Automation tests

For the generic bugs, I think the better way to do is introduce better automation test to our project (not all project got automation test and the test cover is also not enough)

This is absolute must! Without project-wide automation tests it will be snowball.

4. QA-team Need to do better than now :smiley:

---

I agree with @felixonmars being an active tester for almost all known Deepin distros in 90% of all cases bugs / issues are universal, it's relatively rare case to have Arch / Manjaro specific.

With very notable exception of complete lack of GTK theme on Arch / Manjaro which in my mind is major visual / workflow issue now compared to original distro: https://github.com/linuxdeepin/developer-center/issues/613 https://github.com/linuxdeepin/developer-center/issues/930

---

if manjaro want a better tested version for packaging, probably you may need only do packaging after Deepin got update (which means a single release cycle is done)?

A B S O L U T E L Y ! @BLumia @oberon-manjaro You need to figure out a way for Manjaro to get only full release cycle stable updates. At least for now, otherwise we can't really move on, or even create new .iso

[BLumia]

Therefore i assume, that maybe unit tests should also be done again on whole tagged release

We do unit test but it is not full scale test. Full scale test always happends before release since it will spend to much time to do, it is a lots of work to do.

just at least for 5 minutes launch program

That won't solve the problem. Actually we always do some manual test before each time we push a commit, but since it's not possible to do integration test manually each time we commit (in just 5 mins), so it is still possible that one commit breaks somewhere the test doesn't cover.

if manjaro want a better tested version for packaging, probably you may need only do packaging after Deepin got update (which means a single release cycle is done)?

A B S O L U T E L Y !

Just want to know, did Manjaro got a testing channel to test the packages before push update to user?

[keybreak]

Full scale test always happends before release since it will spend to much time to do, it is a lots of work to do.

Well...something must be improved there anyway.

but since it's not possible to do integration test manually each time we commit (in just 5 mins), so it is still possible that one commit breaks somewhere the test doesn't cover.

That's a big problem i see, maybe complete code freeze / lockdown for integration test period? Oh crap, i forgot we're talking about individual programs here...So that's not really an option

Just want to know, did Manjaro got a testing channel to test the packages before push update to user?

Manjaro have:

1. Unstable branch - basically synced to Arch + some custom Kernels / systemd / mhwd patches etc
2. Testing branch - active testing of all packages system-wide, fixing errors making preparations for stable release
3. Stable branch - this suppose to be stable, but in reality (at least with Deepin) now it's not, for example: Prior latest stable update we already knew about serious bugs like 1 (constant dock / settings crashes & coredumps) here.

Yet still @oberon-manjaro had to update anyway, because of some future issues with dependency hell / Kernels / gcc etc.

Well I can not hold back whole stable update for deepin packages with their more or less known or fixed issues and it's extremely difficult to pick out individual packages or groups of packages. We might run into quite a chaos with dependencies. For a lot of other reasons packages now had to be moved on, sorry.

(c) https://forum.manjaro.org/t/stable-update-2019-04-20-nvidia-kernels-deepin/84064/19

As you can imagine...It's not very pleasant experience to have crashing dock / settings on production system (and we'll most likely have this at least for week-two until next stable update, maybe even more unless something will be fast-tracked from unstable which i doubt).

---

So technically yes we have testing branch, but in case of Deepin right now it's practically useless, because it picks a lot of what you reference as (2) Code freeze and then it lands on Manjaro Stable.

---

@felixonmars btw, i was thinking about something Arch / Manjaro specific, even though i'm not user of this feature i've noticed that onscreen keyboard doesn't work / have no settings, is this known?

[keybreak]

@BLumia @oberon-manjaro Any updates on this topic?

Manjaro Deepin Stable is seriously failed this time (already for 10 days):

1. Dock / Settings crashes / coredumps constantly
2. On 4k screen it's unusable, scaling don't work at all in qt and system-wide is very skewed
3. etc etc i could go on.

This is serious, especially considering that all of this bugs was already fixed in Arch for days.

Please, any updates on the situation and how to avoid it, @oberon-manjaro maybe it's time to say at least something (and probably fix it ASAP for Manjaro Deepin Stable as it should be from day 1 of release?), and then you and @BLumia can figure out how to get only full-cycle tested Deepin packages for now to avoid such bs in near future until Deepin get really stable?

[BLumia]

Please, any updates on the situation and how to avoid it

As I said before, if manjaro want a better tested version for packaging, probably you may need only do packaging after Deepin got update (which means a single release cycle is done)?

Another thing is, as you said:

Stable branch - this suppose to be stable, but in reality (at least with Deepin) now it's not

It is you guys let the packages which you said is unstable, into the stable branch, if you really think it's unstable, why it can be in the stable branch, did the testing branch really did the testing job?

1. Dock / Settings crashes / coredumps constantly
2. On 4k screen it's unusable, scaling don't work at all in qt and system-wide is very skewed
3. etc etc i could go on.

I don't think the first and second one is a package distribution issue, more likely generic bugs, so consider fire a bug once you got these issues.

[oberon-manjaro]

The situation with Deepin packages has been extremely difficult, near to impossible, to handle in a sane way from the Manjaro workflow perspective lately. Reason is that package updates appear to be continuously trickling in not in any kind of a congruent or bundled manner with apparent package fixes and an understandable aim of getting to a stable state every once in a while. We have a situation where while on one end problems get fixed, the next new packages with new flaws have already arrived in a different corner. At the same time of course packages depend and rely on each other. So for me there is at the moment no way to combine reliably working packages in one set of updates that could be pushed to one or the other Manjaro branch. My only choice is to hold back all the Deepin packages or move on all of them. On top of that dependencies of compiler or qt5 versions need to match. So I don't see a way to organize this better on my end...

[keybreak]

Both of you have your truth, but we need to find common ground and here's what i see:

1. @BLumia tag somehow fully tested bundle of all stable and tested release packages and avoid mismatch of build dependencies like qt5.

2. @oberon-manjaro at least for now use only this packages for stable (unless something really hardcore happens and overlooked with real need of hotfix & patch)

Otherwise we're doomed to fail.

---

@BLumia other than that of course what was stated before stands and also needs to be addressed to avoid issues in the first place, like whole project automation tests, better overall QA testing etc...

---

@oberon-manjaro

It is you guys let the packages which you said is unstable, into the stable branch, if you really think it's unstable, why it can be in the stable branch, did the testing branch really did the testing job?

I understand why you've done it this way... But still you can't deny that he's right and it's very bad practice with any package even outside of Deepin...At least it's food for thought on improving testing / update process of Manjaro Stable branch, because you never know when stuff like that can happen, and it certainly will happen in this day & age.

---

I don't think the first and second one is a package distribution issue, more likely generic bugs, so consider fire a bug once you got these issues.

Sure - i always report bug when i see one, but like i've said those were long fixed in upstream after ((2) Code freeze), i was just referencing of current problems caused by last Manjaro Deepin stable update (this was mainly for @oberon-manjaro ), and that we need to do something about it like now.

[BLumia]

For generic bugs which is related to source code distribution (but not a Manjaro specific issue), maybe we should pick-up IRC or gitter or Matrix or Discord so we can make the issue process much faster? cc. @hualet

But anyway we are not talking about generic bugs, so let's just focus on the Manjaro specific issue. Only use the tag version after Deepin got a release will let you always get a well tested tag for packaging (and also without dependencies issue about DDE itself). It can still have problem like Qt version and etc, other ports like Debian / Fedora / openSUSE will also get similar issues and you guys can (and it is recommended to) open a new issue for these cases, you should avoid the package got into the stable branch before this sort of issue solved, too.

[oberon-manjaro]

As an example: I am right now about to release the next Manjaro Stable Update. All Deepin packages that I have received from upstream are now collected on testing branch for that. Just yesterday I have forwarded a few of the fresh packages more or less blindly from unstable to testing in order to have a congruent set of packages and assuming that when I see three updates for a package arrive on a daily basis that they include bugfixes and improvements anyway, rather than new breakages... Now this morning, more packages have arrived. They are now again on the unstable branch... It is totally out of my scope to investigate all these package updates individually to find out how to treat them/hold/back/test more/fast forward. For all other DEs like Cinnamon, Gnome, xfce and KDE packages arrive in bundles with additional bugfix releases shortly afterwards. This makes it totally possible to handle but something like this!? Have a look! It's just a constant unstructured stream of updates for me with now way to draw any lines for useful snapshots:

[BLumia]

Well, since we currently doesn't have a platform to query package version in Deepin's package repository, maybe we should make our Code Release Platform public accessible so other package maintainer who do port DDE packages to other distro can use that to track the tag version which is included in the current Deepin release?

@hualet

[hualet]

@BLumia, Downstream maintainers don't need to know our Code Release Platform and they won't I think. The problem @oberon-manjaro complaining here is that we release too many tags (of the same repository) in a short term which indeed is strange compared to other DE upstream.

I'm considering that we may "hide" those internal tags from the public, and push the final ones at once when it's considered stable enough internally?

[BLumia]

The problem @oberon-manjaro complaining here is that we release too many tags (of the same repository) in a short term which indeed is strange compared to other DE upstream.

So one way I think it may works is get a place to let user query the package version, currently CRP got a list of tags which we are used for release(that's why I mentioned the CRP site), actually it will be better if we got a https://packages.debian.org/ similar site so anyone could query the package/tag version without install Deepin (so they will know which tag is the right tag they want).

Or if you have any better/working idea?

[hualet]

The problem @oberon-manjaro complaining here is that we release too many tags (of the same repository) in a short term which indeed is strange compared to other DE upstream.

So one way I think it may works is get a place to let user query the package version, currently CRP got a list of tags which we are used for release(that's why I mentioned the CRP site), actually it will be better if we got a https://packages.debian.org/ similar site so anyone could query the package/tag version without install Deepin (so they will know which tag is the right tag they want).

That sounds good, but it doesn't solve this specific problem. The problem here is downstream packagers don't know when it's proper time to package and release, will there be a new tag tomorrow? they don't know about that, and we don't know either.

Or if you have any better/working idea?

As I said in my previous reply, I think we need to stop making so many tags (for one project) in a single release. We can store internal tags in CRP but not actually put them in the git repository, and use those tags to do packaging and testing before the release. After it's all set, we release the updates and tags(maybe only the latest ones), so downstream can get all the update at once.

[BLumia]

That sounds good, but it doesn't solve this specific problem. The problem here is downstream packagers don't know when it's proper time to package and release, will there be a new tag tomorrow? they don't know about that, and we don't know either.

For Manjaro, just follow the Deepin official package repository (not the newest tags) will be fine. For example, you can got a list of tags which we used to release 15.10. Even if a new tag comes after 15.10 release, downstream packagers doesn't need to package that new one unless that tag went inside the Deepin official package repository.

As I said in my previous reply, I think we need to stop making so many tags (for one project) in a single release. We can store internal tags in CRP but not actually put them in the git repository, and use those tags to do packaging and testing before the release. After it's all set, we release the updates and tags(maybe only the latest ones), so downstream can get all the update at once.

Yep, using commit hash instead of tag can also be good (but Arch will no longer got the more up-to-date version of DDE softwares, too). Or maybe we can mark the tag used for testing with pre-release label (GitHub feature)?

[oberon-manjaro]

For Manjaro, just follow the Deepin official package repository

It's not feasible for me to package everything Deepin related exclusively for Manjaro. We currently rely on upstream Archlinux packaging like with other DEs, except for some adjustments, fast-forwards, hold-backs and some exclusive features and customization here and there. Of course it would help to adopt a packaging policy liike that between Arch stable and Arch testing - but that's beyond me of course. This would have to be decided by @felixonmars

[oberon-manjaro]

I do wonder how Archlinux users deal with the situation, since you have to assume that, as a rule of thumb, what breaks in Manjaro stable must have been broken in Archlinux at least for about a week or so... :wink:

[antechdesigns]

Of course it would help to adopt a packaging policy liike that between Arch stable and Arch testing - but that's beyond me of course.

I have been saying this for months, I even offered to test before release to production machines.

I do wonder how Archlinux users deal with the situation, since you have to assume that, as a rule of thumb, what breaks in Manjaro stable must have been broken in Archlinux at least for about a week or so... wink

Its a nightmare to be honest, I am waiting to create a new ISO but at the moment there are just to many bugs, old & new, releasing one in this state would be an embarrassment.

Unless something changes soon I may have to move on to a more stable project :(

[jnylen]

@oberon-manjaro,

JFYI as a Arch Deepin user. I haven't had any major issues like coredumps since beginning of 2018. And I update packages daily.

But it does feel a bit annoying when some bugs that for me as a user feels like they should've been caught in development or in unit testing.

@BLumia,

I feel like using a pre-release tag on unstable releases would be quite good then arch can have both in their repo if a user wants to do testing.

[jonathonf]

I'm considering that we may "hide" those internal tags from the public, and push the final ones at once when it's considered stable enough internally?

It doesn't matter if the tags are public as long as release tags are identified as such, whether that's through a

pre-release label

or a versioning scheme, e.g. 1.0 -> 1.1dev1 -> 1.1rc1 -> 1.1, or an odd-even stable-development pattern (like GIMP and Xfce, e.g. 1.0, 1.2, 1.4 are stable series and 1.1, 1.3, 1.5 are development series).

Alternatively, don't tag things with version numbers which aren't ready for release. If you release a 1.0 and then tag a 1.0.1, then the implication is that 1.0.1 is a bugfix release for 1.0. If 1.0.1 is not a bugfix release then it shouldn't be tagged as 1.0.1.

[BLumia]

@jnylen : I feel like using a pre-release tag on unstable releases would be quite good then arch can have both in their repo if a user wants to do testing.

Seems good but the pre-release tag seems a GitHub only issue, will looking into it later

@jonathonf: 1.0 -> 1.1dev1 -> 1.1rc1 -> 1.1,

This seems better, will also take into consider, thanks for the idea!

---

btw just make it clearly, both of these could not avoid bugs which our test is uncovered, so other distro maintainer may still need to hold the package and fire a bug to us if any distro/enviroment specific unexpected bug happened.

[es20490446e]

What I do is not to rely on unit testing for detecting bugs, but rely on process. That is:

• Unit test only the most basic functionalities, only to make sure that the software at least runs. So you save having to write plenty of tests that anyway are more reliably done manually.

• Make functions as short as possible, and test each of them as soon as possible before continuing. So you are always building your next block on stable.

• Commit only the code after testing it yourself, in every possible aspect. Don't rely on later discovery, as that takes way more work to fix.

• Only code more functionalities if the current ones don't have bugs. So the work you put translates into finished value as soon as possible.

• Release new code instantly, so its limitations are discovered as soon as possible. Otherwise all the bugs will come together. Have a stable base, with a small amount of beta.

• And the most important one: your code quality strongly depends on how easy is to change things, and publish them. Make changing things as straightforward as possible, it's your best investment. Automate simple repetitive task, and figure out how to make changeovers as fast as possible. Use kaizen to archieve instant changeovers.

[BLumia]

Hi all and here are some update about this topic.

As I mentioned before, our release cycle workflow was tag and test and then finally tag and release the product. Recently this workflow is changed with a new revamped (internal) Code Release Platform(CRP) (by @electricface) which introduced a pseudo-tag feature which allow developer mark a git hash as a test-ready version and it will not release a new tag to the public before QA peroid is done.

Although currently the CRP system is still not available to public, but pretty sure package maintainer for other distro will benefit from this feature :)

Also, the CRP system may be public accessable in the future, btw(no promise now).

[justforlxz]

If you want a more stable version, you should follow the deepin version instead of using the released tag immediately, because we do not guarantee that every tag will work stably. We will consider improving the workflow to improve this issue.