search

veekun / spline-pokedex

heart and soul of veekun's pokédex
http://bugs.veekun.com/projects/spline-pokedex
MIT License
14 stars 7 forks source link

Port to pyramid #115

Open magical opened 6 years ago

magical commented 6 years ago

This is a meta issue to track plans and progress for porting spline-pokedex to pyramid.

Spline

Spline-pokedex is currently built as a spline plugin. My goal is to convert it into a standalone web app. I don't want to port spline to pyramid (that's veekun/spline#34). Besides, eevee already did that https://github.com/eevee/spline/

What does spline actually provide? Most of the code can be divided into two broad categories:

  1. plugin support. finding plugins; integrating with pylons; setting up mako template paths; hooks to allow plugins to override things. we don't need any of this.

  2. helper code. useful helpers for web apps. basic index and error pages, basic styling, flashes, wtforms helpers, markdown support, sql logging. we'll probably want to copy some of this into spline-pokedex.

Question: do we want to merge the spline history into this repo?

Veekun

The veekun repo houses "bits and pieces" of the site. Notably, all the styling that makes veekun look like veekun, and some content pages. It gets included in spline-pokedex via some hacky local-plugin mechanism in spline.

I definitely want to merge the styles and header/footer bits into spline-pokedex. Some of the content, like the dex-history, could arguably live elsewhere, but, eh, might as well drag it in.

GTS

The GTS plugin is old and broken, and will likely never be fixed (#72). We should probably just delete it.

Mako

Pyramid has support for mako. Thank god.

Frontpage

Veekun's front page is powered by the spline-frontpage plugin. Ideally i'd like to keep that working, but i'm not sure if it's worth the effort to port, or if it should just be rewritten. This also ties into the question of whether to merge with spline or not.

i18n

One of the things spline provides (with help from Pylons) is internationalization support. I don't think it will be too hard to adapt to using pyramid's i18n support. Not having to deal with plugins should make things vastly simpler

Routing

Pyramid and Pylons have different routing systems. Pylons uses routes, which is based around a controller/action model. Pyramid has its own system which is built around views

Pyramid also has a traversal-based routing system that you can use, which would probably be a good fit for the pokedex... but it's also different from what we're doing now, and would be more effort to switch to, so for now I intend to stick with the old pattern-based routing system.

Views

The biggest difference between pyramid and Pylons is probably the view system. Also, pyramid doesn't use threadlocal objects for things like the request and template context, unlike Pylons, which uses them all over the place. So we'll have to make sure that we're threading the context through explicitly to every function that needs it. This is probably what will cause the most strife during porting.

magical commented 6 years ago

Here's my overall plan for doing the port. There are actually several steps that we can do while we're still using spline.

Prep work

Pyramid port begins

magical commented 6 years ago

I ported the lookup page to pyramid as a prototype. I think i should also try one of the smaller dex pages (maybe abilities) before doing this for real.

I turns out we don't need much of spline, so i don't think it's worth merging the history into spline-pokedex.

magical commented 6 years ago

I ported the abilities page today (https://github.com/veekun/spline-pokedex/commit/5e9d0142d280853890cb23b9c1a1d2b4c4f09eb0). Was pretty straightforward. Required minimal changes to the controller and template. The main difficulty was updating references to the helper functions that i moved, and fixing up one that i had broken during the move. (https://github.com/veekun/spline-pokedex/commit/47ff94fc360827360c3a9e67155e1172c05bc16b)

magical commented 6 years ago

I intend to pick this up again sometime in the next few weeks.

@eevee, any thoughts?

eevee commented 6 years ago

I think this would be fantastic, and of course more immediately helpful than veekun-pokedex which is a long ways from being anything. Maybe it'd also give me an excuse to run through and spruce up basically everything one bit at a time.

veekun-pokedex does have a couple niceties: the move table (so far) is built from objects and is much less haphazard in general, and the routing is done via resources which is kind of cool but may or may not be more trouble than its worth. Also it has language right in the URL, which I understand to be the Right Way to go about that.

eevee commented 6 years ago

Looking through the pages you ported and I'm pleasantly surprised with how straightforward it is. I guess with the template and database libs the same, the only changes are at boundaries, where we really don't do a lot.

I suppose we should get rid of the c references eventually, and I want to say there's something much simpler we could be doing with SQLA with regards to eagerloading, though now offhand I can't remember what it was. I think around 1.1 they added a way to do explicitly something we've been doing implicitly? Separate query by id, maybe? We should definitely be more consistent about it; so far I've always just looked at the queries and tacked on eagerloads until the overall count becomes reasonable.

(Or there's always the YAML thing, which someday might actually exist.)

Pyramid does have a lot more in the way of plugin APIs compared to Pylons, so I suspect caching won't be too hard.

And, yeah, forget spline. Failed experiment. New spline is barely even the same thing and effectively only powers one website anyway, again.

magical commented 6 years ago

veekun-pokedex does have a couple niceties: [...]

It does. Once the initial port is complete we can start pulling things piecemeal from veekun-pokedex.

I suppose we should get rid of the c references eventually

Definitely.

magical commented 6 years ago

Are you fine with merging the veekun and spline-pokedex repos?

eevee commented 6 years ago

But... but what if someone wants to run exactly my same website with different branding?!

(yeah go ahead, this spline will be dead after the port anyway, might as well break a few eggs)

magical commented 6 years ago

Users/Forum

I guess i forgot mention this in the first post, but spline also provides the users and forum plugins. I don't want to reimplement those, so my intention is to let them go away. The forum is dead anyway, and the login system is based on WebFinger/BrowserID, so i doubt anyone can even log in any more.

I did grab an archive of all the posts with wget, for whatever that's worth.

magical commented 6 years ago

So, i kind of made this more stressful than it needed to be by trying to separate things cleanly into "pre-merge" and "post-merge" steps. The problem is that — as i discovered during my first prototype attempt — there are lots of unexpected issues that can arise during the porting process. Things that could be addressed pre-merge (if i knew about them) but i can't discover until i'm deep in post-merge. And i don't know what they are because i haven't completed a full port yet.

The new plan is to just create a branch, do the merge with veekun, and try to complete as much of the port as possible. Then after everything is working, i'll see if i can factor it into a more reasonable series of commits (or maybe just not bother and merge it as-is). If i find anything that i can cherry-pick to the master branch, i'll probably do that on a case-by-case basis as i go.

That's happening here: https://github.com/magical/spline-pokedex/tree/pyramid_pokedex

magical commented 6 years ago

I'm tracking my progress using the Projects tab, which is actually fairly slick: https://github.com/veekun/spline-pokedex/projects/1

magical commented 5 years ago

All pages have been ported (except error pages). Tests have mostly been ported.

magical commented 5 years ago

I'm picking this up again. The plan is to continue to plug away at the pyramid_pokedex branch until it reaches feature parity with the current version. Meanwhile, i'll backport as much as I can to master.

magical commented 5 years ago

Created the pyramid-backports branch: https://github.com/veekun/spline-pokedex/compare/pyramid-backports. It contains changes that can be safely backported to the spline-based code. The goal here is to minimize the difference between the current code and the eventual pyramid code.

magical commented 5 years ago

Feels like i'm getting close!

magical commented 4 years ago

I want to finish this up. As of the last time i touched it (in mid 2019) it was pretty much complete. I'm sure there are still some unpolished areas, and there are some open questions, but nothing major imo.

I think the next step that needs to happen is to deploy a test instance. Then we can evaluate it and decide how to proceed.

Open questions:

eevee commented 4 years ago

write a new frontpage or port the old one?

The old one is... not... great? The Bulbanews feed doesn't seem to work any more, my blog isn't especially relevant, and the changelog dates back to gitweb days (whereas now GitHub is much easier to read, if you care to). The only other bit of interest is the forums, which no one uses.

It could probably stand to be full of stuff that's actually relevant to the site — a big friendly index (rather than forcing new users to peer through menus), a bit more about the project itself and its status, contributors and how to find them, link to the Discord, etc.

figure out what to do with icons. spline contains complete copies of the famfamfam flags and Fugue icon sets. i'm not sure whether to copy all of them to spline-pokedex, copy just the ones needed, or something else. (n.b. this doesn't really matter)

Could... we... just use flag emoji, now?

Offhand I'm not sure how much of fugue we're actually using, outside of the forums.

pyramid has no built-in way to send email reports of crashes. we'll want to find some way to replace that feature

Probably not that service I accidentally spent over a grand on after they quietly jacked up the price to like $60 a month.

I see a pyramid_errmail but it's quite a bit old. Kinda surprised I don't see anything obvious for doing this. Or maybe people noticed that receiving tens of thousands of emails is not the ideal way to deal with errors.

Could log them locally, and write a thing to send them to Discord...?

should we actually merge into spline-pokedex, or leave this repo as a historical record and create a new repo?

If this is a single consolidated codebase without all the plugin nonsense, then I'd say create a new one. This mess is confusing enough. :) It's what I was doing with my own Pyramid from-scratch experiment (veekun-pokedex), too. We can figure out how to migrate issues later.

magical commented 4 years ago

Definitely agree that the frontage needs a redesign, but since @hugopeixoto just ported the old one, this is no longer standing in the way of the port 🎉

P.S. afaict Bulbanews still works, they just... haven't... posted anything in a while. Saw something on Twitter that they were looking for new writers.

magical commented 4 years ago

Could... we... just use flag emoji, now?

Oh. Possibly. The icons are probably slightly nicer for screenreaders, since we can give them useful alt text.

Offhand I'm not sure how much of fugue we're actually using, outside of the forums.

We don't use much of it, but there are definitely a few sprinkled around the site. I don't really want to audit every file to track down which ones we're using though. (The frontpage, for example, uses icons which are specified in the ini file)

Pyramid has a nifty feature where you can serve files from other eggs. I was thinking it might be nice to throw a famfamfam-icons project up on pypi and use that. Although that would probably complicate serving them from nginx in production.

magical commented 4 years ago

I see a pyramid_errmail but it's quite a bit old.

No commits since 2015, hmm. I do see that its dependency, pyramid_mailer was touched somewhat more recently. Might be a case of "it's stable, why would we change anything?"

eevee commented 4 years ago

Tada: https://beta.veekun.com/

Freshly provisioned Linode, even! Gonna gradually migrate stuff over because I love inventing work for myself.

magical commented 3 years ago

should we actually merge into spline-pokedex, or leave this repo as a historical record and create a new repo?

proposal:

  1. fork spline-pokedex to zzz-spline pokedex
  2. merge pyramid branch into spline-pokedex
  3. rename spline-pokedex to pyramid_pokedex