Closed danrademacher closed 7 months ago
Wait is it as simple as:
git diff 11a39b8b05d318357d3a3b284aac2664b58ef721 f2a386c6b591f9ecf7f21ed1bdb2bd5633b1d2df > jan2021-ohmwebsite.patch
then the harder part:
Step 3 is probably where it could be 1 hour or 100 hours...
Bumping this issue.
We are now further behind upstream and this is beginning to hurt. The really big upstream change that we are missing now is support for JSON in the OSM API. Newer versions of most tools now use the new JSON APIs in some way or another: Tasking Manager, iD, etc. This is going to start hitting us more and more in terms of compatibility with OSM tools.
I really like this patch approach @danrademacher . We have spent some time updating the osm-website
version in osm-seed
to latest upstream, so we have a working Dockerfile for some of the updates to the rails pipeline, things like that.
Just to reiterate your idea here @danrademacher - we should just checkout latest upstream and apply all the OHM specific changes as a patch. This seems like it would be several hours of work, and most of it is frontend / essentially figuring out applying all the same frontend changes to an "updated" version of the osm-website code. I'm not sure how much the frontend code on osm-website has changed, so I am not sure how much work this would be, but this would be my ideal / preferred approach. The patch
file that we develop can also then be maintained separately, in theory making future updates to upstream also much easier.
Of course, if the frontend code has changed a lot, this could be a bit of a rabbit hole - I still believe this is the correct approach, even if it means having to re-wire bits of the frontend.
@danrademacher @jeffreyameyer is it possible to bump the priority of this? Happy to help in any way, though most of the heavy lifting here seems to be on @danrademacher and team, since our changes are mostly frontend.
cc @geohacker @Rub21 @willemarcel
Just also on this patch
approach: it doesn't necessarily need to involve anything fancy - with that list of files, it seems like one could also manually just go through each file / replace with the OHM forked version / see what breaks, etc. It maybe interesting to think through the approach / tooling to do this merge - we will need to get a bit creative, but I feel like our changes are contained enough and just being able to see "the file as it was on osm-website before we forked", "the file in our fork", "the file in current osm-website" and diffs between the three somewhat nicely, should make this doable. Of course, this is all contingent on osm-website not having completely changed their frontend, but I don't believe that they have.
@batpad I saw that Andy Alan added bootstrap to the frontend around 2020. In terms of JS code, I don't believe we have too many changes.
What if, instead of a PATCH file, we always rebase the repository with upstream and squash our changes in order to have all our changes in only one commit on top of upstream?
Wille, that sounds promising!
Regarding patches, I'm not sure the thinking in this issue was really all that fleshed out. I am happy to bump this up priority list, and I think there's going to be work here for both DevSeed and GreenInfo
We have this unfinished PR from Sajjad from Nov 2021: https://github.com/OpenHistoricalMap/ohm-website/pull/63, but I think that we would not use.
To take the patch approach, we would run this updated diff command into a patch file:
git diff 11a39b8b05d318357d3a3b284aac2664b58ef721 356ac0a13739888073804b190c533494f8742f1d > apr2022ohm-website-upstream.patch
The main benefit of the patch approach as I understand it is we are focused on just getting all our code applied to the latest OSM code, not actually trying to preserve the commit history that got us where we are.
That said, I realize I am not quite sure about the precise next steps:
I tried these steps using
Checked out current OHM-staging and did:
ohm-website % git diff 11a39b8b05d318357d3a3b284aac2664b58ef721 356ac0a13739888073804b190c533494f8742f1d > apr2022ohm-website-upstream.patch
Then switched to https://github.com/OpenHistoricalMap/ohm-website/pull/63 which is the same as https://github.com/OpenHistoricalMap/ohm-website/compare/staging...openstreetmap:master
And did
ohm-website % git apply apr2022ohm-website-upstream.patch
apr2022ohm-website-upstream.patch:83: trailing whitespace.
If you feel you have been falsely or unfairly accused of violating this Code of Conduct, you should notify OpenHistoricalMap with a concise description of your grievance. Your grievance will be handled in accordance with our existing governing policies.
apr2022ohm-website-upstream.patch:99: trailing whitespace.
The Citizen Code of Conduct is distributed by [Stumptown Syndicate](http://stumptownsyndicate.org) under a [Creative Commons Attribution-ShareAlike license](http://creativecommons.org/licenses/by-sa/3.0/).
apr2022ohm-website-upstream.patch:141: trailing whitespace.
libxml2-dev \
apr2022ohm-website-upstream.patch:206: trailing whitespace.
RUN vendorer
apr2022ohm-website-upstream.patch:513: trailing whitespace.
error: patch failed: .gitignore:1
error: .gitignore: patch does not apply
error: Dockerfile: already exists in working directory
error: patch failed: Gemfile:3
error: Gemfile: patch does not apply
error: patch failed: Gemfile.lock:62
error: Gemfile.lock: patch does not apply
error: patch failed: Vendorfile:20
error: Vendorfile: patch does not apply
error: cannot apply binary patch to 'app/assets/favicons/ohm_logo.ico' without full index line
error: app/assets/favicons/ohm_logo.ico: patch does not apply
error: cannot apply binary patch to 'app/assets/images/about/osm copy.png' without full index line
error: app/assets/images/about/osm copy.png: patch does not apply
error: cannot apply binary patch to 'app/assets/images/attribution_example copy.png' without full index line
error: app/assets/images/attribution_example copy.png: patch does not apply
error: cannot apply binary patch to 'app/assets/images/ohm_logo.png' without full index line
error: app/assets/images/ohm_logo.png: patch does not apply
error: cannot apply binary patch to 'app/assets/images/ohm_logo_256.png' without full index line
error: app/assets/images/ohm_logo_256.png: patch does not apply
error: cannot apply binary patch to 'app/assets/images/ohm_logo_30.png' without full index line
error: app/assets/images/ohm_logo_30.png: patch does not apply
error: patch failed: app/assets/javascripts/application.js:6
error: app/assets/javascripts/application.js: patch does not apply
error: patch failed: app/assets/javascripts/embed.js.erb:31
error: app/assets/javascripts/embed.js.erb: patch does not apply
error: patch failed: app/assets/javascripts/id.js:5
error: app/assets/javascripts/id.js: patch does not apply
error: patch failed: app/assets/javascripts/index.js:19
error: app/assets/javascripts/index.js: patch does not apply
error: patch failed: app/assets/javascripts/index/changeset.js:10
error: app/assets/javascripts/index/changeset.js: patch does not apply
error: patch failed: app/assets/javascripts/index/directions.js:392
error: app/assets/javascripts/index/directions.js: patch does not apply
error: patch failed: app/assets/javascripts/index/export.js:63
error: app/assets/javascripts/index/export.js: patch does not apply
error: patch failed: app/assets/javascripts/index/history.js:149
error: app/assets/javascripts/index/history.js: patch does not apply
error: patch failed: app/assets/javascripts/index/new_note.js:107
error: app/assets/javascripts/index/new_note.js: patch does not apply
error: patch failed: app/assets/javascripts/index/note.js:46
error: app/assets/javascripts/index/note.js: patch does not apply
error: patch failed: app/assets/javascripts/index/query.js:345
error: app/assets/javascripts/index/query.js: patch does not apply
error: patch failed: app/assets/javascripts/index/search.js:127
error: app/assets/javascripts/index/search.js: patch does not apply
error: patch failed: app/assets/javascripts/leaflet.layers.js:50
error: app/assets/javascripts/leaflet.layers.js: patch does not apply
error: patch failed: app/assets/javascripts/leaflet.map.js:18
error: app/assets/javascripts/leaflet.map.js: patch does not apply
error: patch failed: app/assets/javascripts/router.js:148
error: app/assets/javascripts/router.js: patch does not apply
error: patch failed: app/assets/stylesheets/common.scss:2774
error: app/assets/stylesheets/common.scss: patch does not apply
error: patch failed: app/assets/stylesheets/id.css:1
error: app/assets/stylesheets/id.css: patch does not apply
error: patch failed: app/controllers/users_controller.rb:234
error: app/controllers/users_controller.rb: patch does not apply
error: app/mailers/notifier.rb: No such file or directory
error: patch failed: app/views/export/embed.html.erb:2
error: app/views/export/embed.html.erb: patch does not apply
error: patch failed: app/views/layouts/_head.html.erb:6
error: app/views/layouts/_head.html.erb: patch does not apply
error: patch failed: app/views/layouts/_header.html.erb:2
error: app/views/layouts/_header.html.erb: patch does not apply
error: patch failed: app/views/layouts/error.html.erb:6
error: app/views/layouts/error.html.erb: patch does not apply
error: patch failed: app/views/site/about.html.erb:5
error: app/views/site/about.html.erb: patch does not apply
error: patch failed: app/views/site/help.html.erb:4
error: app/views/site/help.html.erb: patch does not apply
error: patch failed: app/views/site/id.html.erb:17
error: app/views/site/id.html.erb: patch does not apply
error: patch failed: app/views/users/terms.html.erb:25
error: app/views/users/terms.html.erb: patch does not apply
error: patch failed: config/environments/production.rb:23
error: config/environments/production.rb: patch does not apply
error: patch failed: config/initializers/action_mailer.rb:1
error: config/initializers/action_mailer.rb: patch does not apply
error: patch failed: config/initializers/assets.rb:16
error: config/initializers/assets.rb: patch does not apply
error: patch failed: config/locales/en-GB.yml:272
error: config/locales/en-GB.yml: patch does not apply
error: patch failed: config/locales/en.yml:253
error: config/locales/en.yml: patch does not apply
error: docker-compose.yml: already exists in working directory
error: cannot apply binary patch to 'public/styles/sprite.png' without full index line
error: public/styles/sprite.png: patch does not apply
error: cannot apply binary patch to 'public/styles/sprite@2x.png' without full index line
error: public/styles/sprite@2x.png: patch does not apply
error: patch failed: vendor/assets/iD/iD.js:25793
error: vendor/assets/iD/iD.js: patch does not apply
error: vendor/assets/iD/iD/locales/en.json: No such file or directory
After that, I see no commits, which is expected since per the docs, git patch
doesn't actually generate commits. But I also don't see any modified files. The errors above seem to be saying "patch does not apply" even in cases like this where we know we need changes (loading timeline JS):
error: patch failed: app/views/layouts/_header.html.erb:2
error: app/views/layouts/_header.html.erb: patch does not apply
@danrademacher So, what I did:
Created a new branch from ohm-website
staging. Merged in upstream changes to that branch, but always resolving git conflicts to take "our" changes:
git merge -Xours upstream/master
Created a PR against staging
: https://github.com/OpenHistoricalMap/ohm-website/pull/87
Also created a patch file, with this branch compared to openstreetmap/openstreetmap-website
latest with:
git diff upstream/master > ourpatch.txt
ourpatch.txt - this should represent the "delta" between our changes and openstreetmap-website master. There's a bunch of stuff on the Gemfiles and Dockerfile and some other config stuff that I'm pretty sure we should be able to resolve (@Rub21 let's go over these when you have some time).
And then there's a bunch of conflicts in the frontend code, which don't seem to me to be too severe.
It sounds like we need some new Dokcer settings or a new image or something to run locally?
Yes - when we are running against latest upstream, we will need some changes to the Dockerfile to make everything work. What I'd recommend here possibly as a workflow:
That sounds like a good plan!
To make sure I'm clear -- we wouldn't try to apply the patch in any automated way, right? We would pick through manually and resolve conflicts, favoring our OHM code only where we actually made changes to it.
I wonder, after this, could could start pulling in upstream say several times a year and it would be alight lift each time. I think we have been saying that since we first forked!
To make sure I'm clear -- we wouldn't try to apply the patch in any automated way, right? We would pick through manually and resolve conflicts, favoring our OHM code only where we actually made changes to it.
That's correct. Would just require going through each file, seeing what's changed upstream, and if we need to incorporate any of those changes. From a brief look at the frontend conflicts, it seems to me like not that much has changed upstream and the changes should be mostly okay to manage (still a bunch of manual work).
I wonder, after this, could could start pulling in upstream say several times a year and it would be alight lift each time. I think we have been saying that since we first forked!
It would be a good idea to have some sort of "cron" Github Action that would just create a pull request with the latest upstream changes every 2 weeks or so that we deal with.
@batpad @danrademacher , I went through files and made some updates in the ohm-website branch https://github.com/OpenHistoricalMap/ohm-website/tree/merge-osm-website. according to lates the upstream changes, also made updates in ohm-deploy https://github.com/OpenHistoricalMap/ohm-deploy/pull/137 to build the contianer for these changes, Dockerfile works fine for openstreetmap latest version , but there are some issues in ohm-website https://gist.github.com/Rub21/d898480c04822b1fd9b2443856d56744 , all related to the frontend and javascript dependencies, It may take still some time to figure out more about those issues, maybe someone from the frontend could help us on it.
Sorry for the slow reply on this! We can look into the front end issues for sure.
I want to be sure we have fully gotten to same local dev stopping point that you have. Here's what I did:
docker-compose build
and this seemed to work as usualdocker-compose up
-- I thought maybe here I would get similar outputs to yours. Instead, I'm getting an error like this:
web_1 | /var/lib/gems/2.7.0/gems/railties-7.0.3/lib/rails/application/configuration.rb:347:in `database_configuration': Cannot load database configuration: (RuntimeError)
web_1 | Could not load database configuration. No such file - ["config/database.yml"]
I see this file that seems similar but slightly different name: https://github.com/OpenHistoricalMap/ohm-website/blob/merge-osm-website/config/docker.database.yml
Full output of docker-compose up
is here:
https://gist.github.com/danrademacher/58e0f2201a6530d80a94533a709b4f88
@Rub21 I assume you didn't get this error?
I was running in the background this command as usual, which is how we normally connect to database for local dev:
kubectl port-forward service/staging-db 5435:5432 -n default
@danrademacher , could you pull the changes?, I have added a example env vars for Database, the above issue should gone.
@Rub21 I finally got back to this and I am seeing the same error:
Exiting
web_1 | /var/lib/gems/2.7.0/gems/railties-7.0.3/lib/rails/application/configuration.rb:347:in `database_configuration': Cannot load database configuration: (RuntimeError)
web_1 | Could not load database configuration. No such file - ["config/database.yml"]
I wonder if there's some issue of how I am trying to run this locally that/s at the root of the problem.
Here's what I am doing:
kubectl port-forward service/staging-db 5435:5432 -n default
and get the result:
Forwarding from 127.0.0.1:5435 -> 5432
Forwarding from [::1]:5435 -> 5432
ohm-website
repo, I do docker-compose build
, wait till that's done and then docker-compose up
this time, for good measure I also tried docker-compose build --no-cache
, followed by up
but I got the same result.
Full log is here: https://gist.github.com/danrademacher/f6917444eebe2a51fde72d37f0624fb1
@batpad maybe we should find a time to meet up again in person before you leave Oakland and run through this local dev process together to make sure it's working for both of us at least up to the front end issues we need to resolve?
@danrademacher , You don't need to run kubectl port-forward
, I wrote some instruction to start the locally.: https://github.com/OpenHistoricalMap/ohm-deploy/blob/new_web_version/images/README.md, It will deploy its own database with local credentials: https://github.com/OpenHistoricalMap/ohm-deploy/blob/new_web_version/images/.env.example , we don't need to make a copy of the env.example
, the output should look like so : https://gist.github.com/Rub21/9d537cb049bcddd7ced1c6ba84b0cb59
OK, I am testing this now., 13 minutes into the cd images/ | docker-compose build
We'll see if the removal of kubectl port-forward
is actually an improvement here. If it means that every change to website assets locally requires a 30 minute build, instead of our already very slow 10 minute website-only build, then that'll ratchet up our need to get a better way to develop the website locally.
All stuff @batpad and I can go over on Thursday.
OK, report back: The build ran for 35 minutes and then produced an error
=> ERROR [29/42] RUN apt-get update && apt-get install -y libapache2-mod-passenger
Full error log just before and then after the error: https://gist.github.com/danrademacher/5acf1e70225f0b5659a106e2c5a907c2
So different errors than I was getting before, after a longer wait, but still not the same clearly-frontend errors that Ruben found at https://gist.github.com/Rub21/d898480c04822b1fd9b2443856d56744
@danrademacher , I can prepare a guide to make the container a development environment, basically redirect some container volumes to a ohm-website. it may help to you test easily the changes.
@danrademacher @batpad ,I wrote a guide to test the ohm-website in the docker environment,without compiling every change https://github.com/OpenHistoricalMap/ohm-deploy/tree/new_web_version/images#setting-up-ohm-website-for-development-mode
Took a while, but I have gotten this to the point of simple application errors in browser console, as has @batpad
@batpad picking up from Slack, I am now up to Uncaught TypeError: L.OSM.CyclOSM
is not a constructor but I had to do the following:
bundle install
in the container -- this should probably be added to the documentation, but I wish I knew why Yarn was getting GL installed for Sanjay but not for me. The Ruby GEM method is what we were doing before, so seems reasonable to stick with that, at least until we jump to MapLibre.
I have moved the app error fixing over into its own issue. We can keep this issue for the "patch" process, or close it ot in favor of new issues if that's preferred.
riffing off of #422 it re-occured to me that we have to think more strategically about all our forks:
This list is a bit daunting, but really it boils down to OHM Website and iD being our major pain points for upstream activity, which is good.
Other things to keep updated that are version pinned in ohm-deploy
but not forked:
@Rub21 and @batpad will write up a list of what versions we are actually using here at this point for these items
I realize I let this ticket stray a bit from its original purpose: Focusing on a patch approach to ohm-website
particularly and not all the service dependencies. I will move those to a new issue.
Back to the patch approach, here's a command to make the diff between current upstream and our latest commit
git diff 31eb7cc11a524427ca11491fe1e9c98d7afbd2b8 6a279be9df074ce2d26d9f7ee23a44796e681c10 > 20220816upstream.patch
Not quite sure which should be a and which b, but in this case, upstream is the base and we're diffing from that.
That produces a 500K line diff file: 20220816upstream.patch.zip
that cover 538 files.
Of those, 281 are iD-related, which we would ideally overwrite with our own iD when the Rails vendorfile processes.
96 are locales files, which have meaningful changes in them for EN and EN-GB and eventually other languages. But upstream can also include new translation keys, so these are a bit hard to manage.
Then we have ~160 other files.
I tried to use a GH action that would create a PR but that fails, among many reasons because of the conflicts. This action is in this branch https://github.com/OpenHistoricalMap/ohm-website/blob/fork-sync/.github/workflows/sync-fork.yml. I think we should next revisit the patch idea and see if that can be combined with a GH action for regular syncing.
This idea of patch approach is a bit stale. Closing this out
If we look at the Staging branch of
ohm-website
, we are 1700 commits behind OSM upstream: https://github.com/OpenHistoricalMap/ohm-websiteIf we look at the files we have changed since we last caught up with upstream, here's what we've got:
I am not 100% sure all of those changes are still needed -- for example, changes to iD maybe are no longer relevant, since we're running from a separate iD container? And some, like
app/assets/javascripts/ohm.style.js
would not have upstream conflicts since they don't exist upstream.the rest, it seems, would take inspection and then migration to some method of doing these as patches? I'll be hones tthat we've never done that at GreenInfo in any project that I can recall, so it's handwaving on my part.