qcdyx commented 1 year ago

Describe the problem

Now that the validator is available as a web service, we want to consider deprecating the GUI desktop app. We suspect that the desktop app is significantly less used than the web service.

Managing releases for both the desktop app and web service requires more time and resources, and may be a poor use of our time if usage of the desktop app is low.
We want to focus our efforts on what's of the highest value to the community, such as PR review, Mobility Database API, etc.

Proposed solution

To better understand what's the highest value, we need feedback! We want to know

Are you using the desktop app currently? If so, how?
What value is there in having both a desktop app and a web service?
Any special considerations with deprecating the desktop app?
What communication channels are critical to sharing the deprecation?

Alternatives you've considered

No response

Additional context

No response

welcome[bot] commented 1 year ago

Thanks for opening your first issue in this project! If you haven't already, you can join our slack and join the #gtfs-validators channel to meet our awesome community. Come say hi :wave:!

Welcome to the community and thank you for your engagement in open source! :tada:

bdferris-v2 commented 1 year ago

I've argued before and will again argue now that it would be helpful to have a true understanding of the usage of the desktop app before deprecation. I recognize that the maintenance and release process for the desktop app is non-trivial, but I do think there are some potential aspects of the app that make it slightly more useful (e.g. easier to quickly and repeatedly validate a local GTFS feed in non-zipped form when making iterative changes to a feed). Ultimately, if people continue to use it, even with the webapp, it'd be good to dig into why. And if they don't... well, the answer is easy :) But right now I'd argue we don't have a complete picture.

We do know that v4.0.0 of the app had 1808 downloads (Windows=1261, Mac=379, Linux=168). That's not nothing, though I'll note the CLI had 1916 downloads, which is honestly more than I expected. The numbers for v4.1.0 are lower but it's worth noting that it appears no one updated the current release tag at https://github.com/MobilityData/gtfs-validator/wiki/Current-Version that would trigger an update notification for the app. Note: please don't fix that bug until you've read the rest of this comment :)

I know it's not exactly a fair comparison, but what kind of traffic are you seeing on the webapp?

What would potentially make this a fair comparison is if we added simple instrumentation to count the number of times the desktop validator is run. Then it would be a more straightforward comparison against the validation volume of the webapp. This could be something as simple as polling a static URL from within the app when validation is run, where traffic to that URL could be counted over time. (This is not much different than the existing version check we run within the app, though sadly, there doesn't seem to be any way to count traffic to a GitHub Wiki page). I'd be willing to implement this quickly, bump to a v4.1.1 release, tick the Current-Version wiki, and see what happens.

Thoughts?

qcdyx commented 1 year ago

Hello Brain @bdferris-v2 Thank you for your comment. I agree with you that running analytics on the desktop app to assess current usage is essential. Please go ahead with the quick implementation and assign it to me for PR review.

bdferris-v2 commented 1 year ago

Per the discussion in today's meeting, does MobilityData have a web endpoint somewhere that has enough server-side logging to be used for this task? @qcdyx @davidgamez

qcdyx commented 1 year ago

Hello Brian, thank you for your comment. We don't have a ready to use url for now, we are currently working on it. @davidgamez we can discuss it tomorrow or next week and we'll get back to you. @bdferris-v2

qcdyx commented 1 year ago

As per yesterday's discussion, I would like to share some thoughts on Approach 1: tracking just desktop vs. Approach 2: tracking 3 apps (CLI + desktop + web).

Approach 1: Pros: Simplicity: Having a dedicated desktop application for polling the URL with server-side logging simplifies the implementation process. We can focus only on the desktop app’s requirements without the complexity of integrating other platforms.

Control: We can tailor the polling solution specifically for desktop environments.

Cons: Maintenance and Updates: Desktop apps maintenance and updates can be time-consuming and resource-intensive.

Limited Reach: Users who prefer CLI and web app won’t be able to participate in the polling process.

Approach 2: Pros: Wide Reach: Supporting multiple platforms allows a broader audience to participate in the polling process.

Cons: Increased Complexity: Supporting multiple platforms requires additional development effort and maintenance. Each platform has different implementations, adding complexity to the project.

User Privacy: We would need ensure that we handle user data and privacy appropriately as it has a bigger impact on our users than just tracking desktop app. We might need to obtain explicit consent from our users for such tracking.

Performance Impact: Polling a static URL can introduce additional network requests and potentially impact the performance of web validator and CLI.

In conclusion, I would like to propose that we focus on Approach 1 for the moment. @emmambd @davidgamez

qcdyx commented 12 months ago

Current Plan as per today's meeting with David and Fred

Create a new endpoint => CurrentVersion return the same Markdown as Github Wiki
Test locally
Investigate Google cloud logs
Get familiar with GCP and track the number of hits by users

qcdyx commented 11 months ago

Update: we have a current version endpoint locally available and wait for Jarvus to setup QA env to deploy it at the next release. @emmambd

qcdyx commented 11 months ago

The endpoint returns json containing the current version info.

bdferris-v2 commented 11 months ago

Has this endpoint been deployed yet?

emmambd commented 11 months ago

@bdferris-v2 @qcdyx It's waiting on https://github.com/MobilityData/gtfs-validator/pull/1560 which is still in progress. Hoping to merge next week.

emmambd commented 10 months ago

Reopening issue, since the PR only covers capturing analytics.

davidgamez commented 10 months ago

@bdferris-v2 We deployed an HTTPS endpoint on the new staging environment to resolve the validator's version. The production endpoint will be deployed as part of the next release.

Staging address: https://stg-gtfs-validator-web-mbzoxaljzq-ue.a.run.app/version Prod address(not deployed yet ): https://gtfs-validator-web-mbzoxaljzq-ue.a.run.app/version

Current staging response:

{
    "version": "4.1.1-SNAPSHOT"
}

cc: @qcdyx

bdferris-v2 commented 9 months ago

What's the rollout plan for the prod endpoint? I'm also surprised it's not something like https://gtfs-validator.mobilitydata.org/version

davidgamez commented 9 months ago

Hi @bdferris-v2, we will deploy the new prod endpoint when a release is created. I understand your concern regarding the DNS address. I'll work on creating a custom domain that we can have more control over rather than the cloud-run-generated one. Please hold #1586 until we have a new address.

davidgamez commented 9 months ago

Hi @bdferris-v2, we will deploy the new prod endpoint when a release is created. I understand your concern regarding the DNS address. I'll work on creating a custom domain that we can have more control over rather than the cloud-run-generated one. Please hold #1586 until we have a new address.

We added a routing rule to the deployed Load Balancer that allows us to have stables DNS names for the API. Find below the new URLs:

Staging: https://gtfs-validator-staging.mobilitydata.org/api/version Prod: https://gtfs-validator.mobilitydata.org/api/version

I think we should always use Prod as a reference for the latest release, even though the version endpoint is not deployed currently. This is supported by the fact that no released applications don't show the "update version available message". To minimize the network failures on the validator's side, we can merge #1586 close to the release.

bdferris-v2 commented 9 months ago

I've updated #1586 to reference the prod endpoint. The VersionResolver handles a 404 on the version endpoint gracefully enough, so it's probably fine to merge the PR whenever. Version upgrade notifications don't trigger for SNAPSHOT builds either way.

bdferris-v2 commented 9 months ago

1586 is merged. At this point, I think we are ready to cut a release of the desktop app, at least for this effort.

cka-y commented 5 months ago

Task:

[x] Retrieve statistics: how many times the desktop app has been ran in the last month?
[x] Create a dashboard in GCP logging

qcdyx commented 5 months ago

Analytics (2023/8/17 - 2024/2/1)

MobilityData / gtfs-validator

GTFS-Validator Desktop App Deprecation #1520

Describe the problem

Proposed solution

Alternatives you've considered

Additional context

1586 is merged. At this point, I think we are ready to cut a release of the desktop app, at least for this effort.