Only disable third party apps for major upgrades

[PVince81]

See https://github.com/owncloud/core/issues/14026#issuecomment-77671051

Major upgrades as in the new versioning scheme: 8.0 -> 8.1 is a major upgrade, 8.0.0 -> 8.0.1 isn't.

Not sure what can be done for apps that still break minor updates (not sure which, since there should be no API changes in minor upgrades)

@KeMBro2012 @DeepDiver1975 @karlitschek

[PVince81]

I'd say the probability of having apps break during a major upgrade (ex: 8.0->8.1) is much higher due to API changes / library upgrades, etc.

I guess most apps should survive minor upgrades, maybe it's still worth taking the risk not disabling them for minor upgrades.

Admins that use the "stable" or "8.0" repo/channel would expect to be able to update automatically without any further interaction. Admins that use the "testing" or "stable" (the one without fixed version) repo/channel should expect major updates to appear and might require extra interaction. So instead of a config switch it is mostly about which channel was used.

What do you think ?

[oparoz]

The problem might come from various workarounds put in place after a dot zero release. Devs will want their apps to work and maybe they'll introduce changes without making them available only for that specific version (their mistake), which means that after an oC upgrade which fixes a problem, the app might stop working and break oC altogether. Just a theory... :)

[awikatchikaen]

Why not doing like piwigo (anothe php webbased service) and its plugins ?

All plugins have in metadatas the core version compatibility. When the core is upgraded, there is a warning with a list of plugins still not marked as compliant by the maintainer. If the OC admin still do the upgrade, only the not compliant plugins are disabled...

There, for maintainers that have tested during betas, the plugin is already OK when the core release appends and no one is annoyed by the unuseful deactivation. And for the other, the admin is warned, and can wait if it think the plugin is important, and can later check via the upgrade if those plugin have evolved.

[karlitschek]

@awikatchikaen We do the same. App developer can add the information which core versions are compatible. Unfortunately a lot of (most) app developer don´t maintain this information properly in the apps.

[BernhardPosselt]

Another solution would be to not include appinfo/routes.php and appinfo/app.php on any upgrades so no thirdparty app code is executed on upgrade. If an app misbehaves this will leave you with a blank screen but after the upgrade, but the upgrade should have been successful. Also if a user reenables an app he will also be greeted with a blank screen.

This is just an idea, but couldnt we reengineer the behavior of GNOME or other systems when you change your display resolution? They give you a time window of 15 seconds. If you dont hit ok, they will revert to the previous settings. In terms of ownCloud what about we do the following on upgrade:

• disable all apps
• upgrade
• enable one app and create a request to the files, apps and admin page or a page that is usually blank after an error. If these requests dont return successfully we can disable one app with the notice that the app is not working

[awikatchikaen]

@karlitschek : Then, if a screen is displayed just before upgrade with list of non-compatible apps, some users will delay upgrade and pressurise addons developers to update theirs apps and maintain this information before next big upgrade....

[BernhardPosselt]

@awikatchikaen almost no apps add a maximum version so this wont help

[mgriego]

I'm coming into this a bit late in the game, but one comment I would have is that the auto-disabler that's currently on 8.0.2 doesn't seem to log the apps that were disabled anywhere... that seems like a pretty large oversight to me if you're auto-disabling part of someone's configuration.

[DeepDiver1975]

I'm coming into this a bit late in the game, but one comment I would have is that the auto-disabler that's currently on 8.0.2 doesn't seem to log the apps that were disabled anywhere... that seems like a pretty large oversight to me if you're auto-disabling part of someone's configuration.

disabled apps are printed on the terminal in case the console is used - for the web based upgrade the apps are displayed on the screen. Missing piece: https://github.com/owncloud/core/issues/14569

[mgriego]

@DeepDiver1975 Yes, and I saw the apps that were being disabled for a brief moment before the page redirected away. So, #14569 is certainly a step in the right direction. When I got hit by this, I immediately went to the OC log to see what it had disabled, thinking that surely it logged its actions like any well-behaved app..........

[mgriego]

My 2 cents... if OC is to be a cloud for the masses solution, it should warn before doing something like this (despite putting it in the rarely-read docs), and it should be able to tell you what it did after the fact, so you can correct/reconfigure anything that changed. That's just good UX.

[KeithBronstrup]

@mrlego that was my point when I first complained about it. Rather than taking the easy way out and saying "well, other developers can't be assed to maintain theit projects correctly, so we're going to disable all 3rd party apps and break your OwnCloud install for certain, on the off chance that you have an incompatible app that might break it", the correct response (IMO) would be to warn (and give users the option to disable) when an app has a missing (or unrealistic) max version specified, or lacks compatibility data altogether; and warn of this issue on the download page, as well. Things should only be automatically disabled if the compatibility data says they're incompatible; and when something is automatically disabled, it should be logged and the page displaying the notification should not automatically forward. Let the app developers figure out why people stop using their apps; the problem will solve itself if you simply implement (and enforce) a few simple policies to combat it, rather than embracing it and making the upgrade process a huge PITA and causing people to simply stick with old (and vulnerable) versions of OwnCloud.

For all intents and purposes, an OwnCloud installation that loses functionality after an upgrade is just as broken as one that doesn't come back up at all. Someone who is in a hurry might upgrade, see that the page loads afterward, and assume that everything is fine, while it's obvious there's a problem if the page doesn't load after the upgrade; and yes, even the best of us are in a hurry sometimes, so having it simply not load if something is incompatible is preferable in most cases I can think up.

Of course, the issue of administrator competence will be raised in argument to my position. Please, save it. Yes, a competent administrator would hold the upgrade until they had time to fully test it, negating that specific argument; but then, the same competent administrator would know how to move 3rd party apps out of the apps directory and move them back in one at a time until it breaks (thereby identifying the offending app). We who have some semblance of competence do not want ou installs breaking after every upgrade; only after those during which an actual problem occurs (though, preferably, not at all).

Furthermore, I'll reiterate something I said in my original complaint: apps that were limited to certain users or groups before being disabled are no longer limited after being re-enabled. This has to be done manually; and if an app exposes, say, a company's internal data that they, perhaps, don't want clients seeing (we're talking potentially sensitive data here), that's a security vulnerability right there. All in the name of taking a shortcut during the upgrade process and holding the hands of the incompetent as you walk them off a cliff.

[KeithBronstrup]

@karlitschek that's the app developer's problem, then. Do a bit of validation in the apps market, read the app's manifest and if it's invalid or missing some compatibility info, note that on the app's page, or, you know, reject the submission until they manage to submit it correctly. Do some basic quality control on the apps that are offered through the OwnCloud market and provide prominent warnings about lack of support for apps from other sources. By providing a simple way for your users to do their own due diligence to determine whether or not an app will work for them, you shift responsibility of ensuring that their OwnCloud installation doesn't break onto the user, where it belongs; more importantly, by clearly identifying and marking (or, better, rejecting) apps that do not follow compatibility guidelines, you shift much of that responsibility to the app developers who want people using their apps and supporting their work via donations.

@Raydiation's per-app test idea, coupled with not automatically advancing past the upgrade report, would work as a secondary mechanism to ensure that only the apps that actually fail are disabled, and the admin is properly notified of the action.

[PVince81]

Another alternative or next step would be to automatically re-enable the apps that were disabled (and make sure they got reassigned to the proper groups if applicable), and run the upgrade for those apps. This could be done after the core upgrade was successful.

[PVince81]

Note that in my personal experience lately I almost always forgot to reenable the apps because I use "zypper update" to update my ownCloud, which also automagically runs "./occ upgrade". Only after I see weird error popups from KDE/Koragnizer I remember that I forgot to reenable calendar/contacts.

So I'm all for finding a proper solution for this.

[KeithBronstrup]

My read on this issue, from everyone's comments in this thread, is that any solution other than the current practice of disabling all 3rd party apps and leaving them that way is an improvement. I like @PVince81's suggestion but (and I'm not familiar with the OwnCloud codebase, so I may be a bit off-base with this) it seems as though it may be one of the more complicated solutions suggested, implementation-wise. That said, it also seems like it would be the most robust, provided that there is some mechanism by which OwnCloud can tell if an app is working that doesn't rely on the developer of the app following some sort of development structure (if everyone could just do that, this wouldn't be an issue, as OwnCloud could just use the min/max version from the app manifest).

I still think that using the min/max version from the app's manifest (after some sanity checking) and disabling apps which don't specify the data, or don't pass the sanity check, would be the simplest to implement, and would put pressure on app devs to get their ducks in a row. More details in my previous posts, of course.

Even if that option isn't ultimately what gets implemented, I would like to see it discussed here, along with the options posed by @PVince81 and @Raydiation.

Some of us are using OwnCloud professionally and not just internally, but as a customer-facing portal. It's really bad to have it (in the customer's eyes) breaking all the time when we update and have to completely reconfigure apps and permissions.

[PVince81]

Note, for information: it is not technically possible to detect many of the app errors due to the current app architecture. Loading of apps is done with PHP's "require_once" which doesn't throw exceptions if syntax/code errors are detected, unfortunately. See https://github.com/owncloud/core/issues/5508 and @Raydiation's proposals here https://github.com/owncloud/core/issues/8742

[BernhardPosselt]

@PVince81 its possible by simply doing two requests, one which has the app enabled and one which does not load the app but allows you to disable it. That way you can check the http status code and conditionally disable the app

[PVince81]

With "request" are you talking about HTTP requests ? Note that whichever update process must also work on the CLI (./occ upgrade)

[PVince81]

How to move forward ? Defer to 8.2 ?

I'll repeat my suggestion: only disable third party apps for major upgrades. Starting with 8.0 it means upgrading from 8.0->8.1->8.2. That path would disable third party apps, because of possible architectural changes that might or might not break apps. For paths like 8.0.1->8.0.2, keep the third party apps enabled.

I think it could be a good compromise until we get to the core of the issue ?

[PVince81]

So an option has been added on the CLI to upgrade without disabling: https://github.com/owncloud/core/issues/16221

Let's keep this ticket here open to continue the discussion.

[rbu]

Should this option be used in the RPM/DEB packages then? There's a related discussion about automatic package updates in #16806 that could possibly be moved in here or to its own bug.

[rbu]

@RealRancor mentioned in #16806 that disabling add-ins when Upgrading (8.0 -> 8.1) might be a good idea whereas updating (8.0.1->8.0.2) it probably is overkill. How could something like this be achieved in systems with unattended upgrades?

[ghost]

@rbu You're really doing unattended upgrades? From my PoV thats the worst thing you can do. In my opinion an upgrade (in general, not only in ownCloud) should be always controlled by an admin.

Edit

Btw.:

hat disabling add-ins when Upgrading (8.0 -> 8.1) might be a good idea whereas updating (8.0.1->8.0.2) it probably is overkill.

I'm just a user as you are and thats just an opinion / my experience. Maybe others (especially developers) have a completely different PoV on this.

[awikatchikaen]

@RealRancor : Yes, with "apt-get upgrade", I don't always check wich packages are updated ...

[dwt]

Just out of interest: why don't you guys have a pre-release qa period for new software where such incompatibilities are found and fixed?

slightly sarcastic mode on I could even envision that having a build server that automatically checks oc core functionality after every commit (with at least the plugins from oc itself active) could catch all instances of such problems immediately after each commit. In our company at least that would be standard operating procedure. You might try it, it's quite nice. slightly sarcastic mode off

@RealRancor: You do realize that there are distributions like CentOS where you can be assured that maintainers are careful and will only release what you can and should install unattended? (Granted you still need monitoring)

If OC has it's own CentOS Repo (they have) - and since the don't say something like 'This repo has horrible quality, don't use it on production systems' - the assumption is that it will deliver updates that I really want to install and that are checked to the best of the knowledge of the OC team.

[rbu]

@RealRancor, yes, I choose a stable distribution and then run updates automatically during a time of day when I'd actually if things break / need a hand. Quoting from https://wiki.debian.org/UnattendedUpgrades

Its purpose is to keep the computer current with the latest security (and other) updates automatically. If you plan to use it, you should have some means to monitor your systems

Given proper availability monitoring, I prefer servers to maintain their software themselves.

[ghost]

@dwt Core apps are not the problem and there is a CI at https://ci.owncloud.org catching problems in core functionalities. The issue discussed here are 3rdparty apps which are breaking oC. The same is very likely also causing problems by so called "stable" software within repositories where you put in untested 3rdparty addons/applications.

[ghost]

Ah, and btw. Seems that updates and upgrades within ownCloud are mixed here:

Update: OC 8.0.3 -> 8.0.4 Upgrade: OC 8.0.4 -> OC 8.1.0

On an upgrade its currently needed to disable 3rdparty apps to make sure those are not breaking oC and i was talking about not doing such upgrades unattended (Like with software upgrades done within Debian 6 -> Debian 7 or CentOS 6 -> CentOS 7).

The discussion here is about not disabling apps during an update. Then automatic updates should be fine.

Edit

Btw. i don't know why Debian is calling that wiki page and the software unattended-upgrades/Unattended Upgrades. Actually they are doing mostly updates within a stable distro.

[strugee]

Btw. i don't know why Debian is calling that wiki page and the software unattended-upgrades/Unattended Upgrades. Actually they are doing mostly updates within a stable distro.

No, they're not. Not in Debian-land. In Debian terms, an update is where the package manager refreshes its list of packages. An upgrade is where the package manager actually goes out and retrieves new packages and installs them. (A dist-upgrade is, roughly speaking, an upgrade to a newer Debian release.)

Of course, ownCloud has its own terminology. And that's fine. (I will point out, however, that Debian's is much, much, much older.)

[ghost]

Of course, ownCloud has its own terminology.

Thats not an ownCloud specific terminology. :-) It is common in software that an update means to raise a minor version where an upgrade means to raise the major version.

[dwt]

Am 17.06.2015 um 09:50 schrieb RealRancor notifications@github.com:

Of course, ownCloud has its own terminology.

Thats not an ownCloud specific terminology. :-) It is common in software that an update means to raise a minor version where an upgrade means to raise the major version.

Well, I’m doing software for quite some time now, and I have to say, OC is only the second project I found that had such a finely detailed differentiation. If you want, I’d welcome you to do battle with the homebrew guys, they think update means, upgrading the package manager, while upgrade means updating individual packages. Or the other way around.

/me just shakes hishead and tries to work with more sane people when he can

[ghost]

I don't think that it makes much sense to discuss this in here. Fact is this is not an oC terminology and used in a lot of other software projects.

[PVince81]

Raised @Raydiation's suggestion https://github.com/owncloud/core/issues/14754#issuecomment-83960983 as separate ticket: https://github.com/owncloud/core/issues/17435

[PVince81]

@DeepDiver1975 @Peter-Prochaska @VicDeo I'd be tempted to move to backlog for now, but this is likely going to continue happening with market apps. We'll eventually need to look into a safer way of enabling apps and test-upgrading apps in case they break stuff.

[PVince81]

whatever, backlog now...

I hope the market will have a stronger validation for app versions, etc, so maybe at some point we could even keep all apps enabled on update.

[KeithBronstrup]

Crap like this is why I'm spending a few hours this month migrating to a combination of OneDrive and Exchange. As much as I love the idealism of having all of my data under my control, that's not the actuality of the situation with OwnCloud, where I can either choose to remain vulnerable by not installing updates, or choose to lose access to features and data, potentially permanently if some app I'm using can't be re-enabled after the update.

That's not my data under my control, that's my data under the control of a team of people who don't have my needs at heart.

Microsoft, for all their flaws, at least provides a consistent experience; and they don't make it too difficult to back up, either.

At least for my small team, the monthly cost per user is much less than the monthly billable cost lost to maintaining OwnCloud.

Just some food for thought.

-- Keith

On Thu, Apr 6, 2017 at 10:41 AM, Vincent Petry notifications@github.com wrote:

whatever, backlog now...

I hope the market will have a stronger validation for app versions, etc, so maybe at some point we could even keep all apps enabled on update.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/owncloud/core/issues/14754#issuecomment-292251705, or mute the thread https://github.com/notifications/unsubscribe-auth/AEE6EBaiKGekOufrJMWL73KQez9WKJb5ks5rtSOzgaJpZM4DrhsD .

[ghost]

@KeMBro2012 Seems you never have heard from the --no-app-disable command flag to occ which is there since nearly two years: https://github.com/owncloud/core/issues/14754#issuecomment-105494067 😄

Just some additional thoughts on your comment:

IMHO you're not forced to use ownCloud. And if something is not working for you how you're expecting it, it doesn't give you the right to blame other people like "that's my data under the control of a team of people who don't have my needs at heart."

Furthermore ownCloud is Open Source and the beauty of this is that you're free to contribute and fix/implement something which is not working as you're expecting it. You don't have this possibility when using proprietary software like the one of Microsoft.

if some app I'm using can't be re-enabled after the update

So how is this related to ownCloud itself? If an app can't be re-enabled after the update then the app has an issue and needs to be fixed by the app developer. Or what was the issue here and how is that related to this issue here?

[KeithBronstrup]

@kdslkdsaldsal I'm quite aware of that command-line flag, but I am also quite aware that the recommended way of installing and maintaining OwnCloud is via a repository, wherein that command-line flag is useless. Thanks, though 😄

I was in this conversation long before that command-line flag was added to OwnCloud; my last comment here was https://github.com/owncloud/core/issues/14754#issuecomment-90503682 wherein I detail much of what was discussed before that comment. Much like the comment you just replied to, it was also not my first comment here. I tried to be productive, I tried to share ideas, I tried to work with the OwnCloud team toward a solution, and I may even had been willing to spend my own time implementing that solution once the concept had been accepted (more on that later, as it touches on something else you've said here). Where is your useful and productive input, @kdslkdsaldsal?

You're absolutely right, I'm not forced to use OwnCloud, which is why I won't be after this month. However, if something is not working as expected and a project is seeking input, it is my place as a user to make my voice heard. For the duration of time during which I choose to use OwnCloud, yes, my data is very much under the control of the OwnCloud dev team and the people who decide which PRs to accept or reject (more on that last bit later, as it pertains to something else you've said here). Until I migrate to another solution, my choices are to remain vulnerable by not upgrading or risk losing functionality (and possibly data which relies on that functionality in order to be properly utilized) during an upgrade. In fact, I will be losing data in the migration, as some of that data is associated with features added by 3rd-party apps which don't really have a parallel elsewhere. So that data remains under OwnCloud's control, as I can't really take it elsewhere; as a result, I lose that data when I go.

Such is life.

Yes, OwnCloud is Open Source and, in theory, the beauty of that is that I am free to contribute, fix, and implement. The reality, as someone who was once a strong proponent of the Free Software Movement, actually lies somewhere in between that idealism and proprietary software.

If I fix this, I already know my fix won't be accepted, as it has already been decided in this thread that the issue wasn't worth fixing. It was stated that things work the way they do because "reasons" and it was not going to change; any PR I might submit to change that would surely be rejected. So yes, while I could still fix it for myself in that case, I will have, then, added an integration task to every update; I would, then, need to integrate my fix into every update that comes after it, and fully test before deploying. At that point, I'm no longer using OwnCloud, but my own personal fork of the project, and the problem I've stated here is that I'm already spending more time than I can afford maintaining my OwnCloud installation. How does your suggestion that I fix it myself help with that?

So, no, I really don't have any more opportunity, in any reasonably non-pedantic sense of the word, to fix issues in an Open Source project than I do in a Closed Source project; both require that the maintainer agree that the issue is an issue, be willing to see the issue fixed, and be willing to let me do the fixing. Yes, I can fork the Open Source project, which is something I can't do with Closed Source; but, again, how does that fix my problem? It doesn't; it adds to it, because then I don't just have apps to re-enable after an upgrade, I have upstream patches to integrate and test against my modifications, which would be more work and take up more of my time.

There is a reason I chose a pre-existing team-backed project, rather than rolling my own; quite simply, I don't have time for that.

Microsoft might not give me the ability to fix issues and add missing features myself, but they do give me a phone number I can call to have that done on a priority basis if I'm paying for it. I've spend 20+ hours on the phone with them over the past 30 days hashing out implementation details for features and fixes in several Office products, including OneDrive and SharePoint, and those are being implemented.

Yes, I know, you're gonna say "but then you're paying for it". Yes. I am. Much less than the billable hours I would have to lose to have done it myself, as a matter of fact! You see, I've done the math, and it would cost me much more (in lost billable hours) to maintain my own fork of OwnCloud to incorporate the fix they've already said they don't want than Microsoft's solution will ever cost me.

You seem hurt that I'm not contributing my time and code to fix this, but I never heard back from the OwnCloud team on my proposed solution and, thus, never got approval to joint the team and implement that fix. Doing so without first knowing that (pending quality of code, of course) my work would even be accepted simply makes yet more work for me and costs me yet more than I'm already taking issue with.

You do see this, right?

Perhaps, if some consensus could have been reached regarding how to handle this, it could have been fixed. If the OwnCloud team, themselves, weren't interested in being the ones fixing it, they could have encouraged one of us who raised the issue (it wasn't me, by the way; it seems like you're brand new to this thread, so I just thought I'd point that out) to fix it by accepting one of our proposed solutions as "the way forward".

I would have started working on it right then and there. In fact, it would have been a race between at least 3 of us, to see who could build it better, faster.

[ghost]

Huge wall of text, to lazy to answer 😄 . Just two short notes:

I'm quite aware of that command-line flag, but I am also quite aware that the recommended way of installing and maintaining OwnCloud is via a repository, wherein that command-line flag is useless.

Exactly there you can use the command-line flag. The packages from the repositories are not doing the database updates for you so you're free to also use that flag when using the repositories.

The main issue i see here was, that too many people raised there voice and started to discuss stuff not even relevant for the main original issue:

Don't disable 3rdparty apps for minor updates

Some one could have just stepped up in the last two years and just implement that one and i doubt that it would have been rejected (you also don't need to be in the "team" to submit pull requests).

Personally i'm using the --no-app-disable command line flag since its introduction and mention in this thread two years ago without any issue during minor updates. So using that flag / option programmatically during minor updates should have been a complete valid fix for this raised issue here.

[PVince81]

If someone has time to implement a check to only disable for major releases but not minor releases, I'd be willing to accept it.

@DeepDiver1975 will the market have stronger validation to make sure apps have proper deps info and min/max versions ?

Side note: since the fork some things that weren't possible before have become possible, so a decision made two years ago can be reconsidered.

[PVince81]

also in general times change, so past decisions must be questioned because many factors that might have existed before might not exist any more in the present

[KeithBronstrup]

While I am still migrating my data away from OwnCloud, I will still consider assessing my available time to contribute this feature/fix.

I make no promises, as my workload has changed (increased) considerably since the time when I would have jumped on this without a second thought (thus why I am now choosing to pay someone else to manage the details), but I'll do what I can.

Since my issue is primarily a time issue, I hope it is understood that I lacked the time to bring that decision from two years ago into question, as I also now lack the time to immediately begin familiarizing myself with the OwnCloud codebase well enough to quickly and correctly implement this feature/fix. For that reason, I was leaving the discussion up to everyone else.

I now find myself in a position where certain pieces of Microsoft software are the de-facto standard by which I must communicate with a subset of my clients, and a massive chunk (per user) of OneDrive storage is included in the licensing of that software, so the reality is that it solves my time issue at no additional cost (beyond what I'm paying for support in order to have certain features implemented).

At the end of the day, ideology is important, but practicality wins out. While I'd love to see my needs met by Open Source software (and in many areas of my business and personal life, indeed, I do see this), I am considering needs, rather than wants, and I must make the decisions that see those needs best fulfilled.

I'll keep revisiting this until either I find the time to implement the feature/fix, or someone else beats me to it.

On Mon, Apr 10, 2017 at 12:05 AM, Vincent Petry notifications@github.com wrote:

also in general times change, so past decisions must be questioned because many factors that might have existed before might not exist any more in the present

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/owncloud/core/issues/14754#issuecomment-292866603, or mute the thread https://github.com/notifications/unsubscribe-auth/AEE6EFpv-rLh2g452IuzyOsvcUX9ddkPks5rudTSgaJpZM4DrhsD .

[ghost]

@KeMBro2012 FYI: https://github.com/owncloud/core/pull/27730

[PVince81]

10.0 will not disable third party apps any more.

[KeithBronstrup]

@kdslkdsaldsal Thanks, got it.

On Mon, Apr 24, 2017 at 12:16 PM, Vincent Petry notifications@github.com wrote:

Closed #14754 https://github.com/owncloud/core/issues/14754.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/owncloud/core/issues/14754#event-1055400332, or mute the thread https://github.com/notifications/unsubscribe-auth/AEE6EIk4GR8QQQsjkZvwkv1vYSuEQFftks5rzPUYgaJpZM4DrhsD .

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.