search

venliong / crxdoczh

Automatically exported from code.google.com/p/crxdoczh
0 stars 0 forks source link

It's time to refactor changed code in servers #8

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
It's VERY UGLY, MESSY and hard to maintain (it's not synced with the current 
version of server2). And I'm more familiar with server2 and Python than before.

Proposals:
Write CrxdoczhSamplesDataSource, GoogleCodeSubversionFileSystem and import the 
correct classes in server_instance.py.
Write an APIServlet to serve API requests.
Reuse existing AppEngineUrlFetcher, PersistentObjectStore, etc.
Keep changes that's possible to cause merge conflicts as minimal as possible.

Original issue reported on code.google.com by fangjue2...@gmail.com on 6 May 2013 at 1:35

GoogleCodeExporter commented 8 years ago 
There will soon be several major changes to both server2 and templates, 
including:
https://chromiumcodereview.appspot.com/12334016/
https://chromiumcodereview.appspot.com/12996003/
https://chromiumcodereview.appspot.com/14273041/
https://chromiumcodereview.appspot.com/15127007/
and https://chromiumcodereview.appspot.com/15087006/
We have to do this as early as possible to catch up!

Original comment by fangjue2...@gmail.com on 14 May 2013 at 9:52

GoogleCodeExporter commented 8 years ago 
Because server2 now pulls everything in svn into data store, cron jobs will not 
be as flaky as before and can be ran more frequently. I'd also like to let 
master server periodically fetch from slave-docs instead of slave-docs 'push' 
updated docs to master.

Original comment by fangjue2...@gmail.com on 16 May 2013 at 4:07

GoogleCodeExporter commented 8 years ago 
Another idea: cron should be clever enough to skip rendered files that match 
the up-to-date version.

Original comment by fangjue2...@gmail.com on 16 May 2013 at 3:15

GoogleCodeExporter commented 8 years ago 
It's running on slave-docs and master-mirror. There are many TODOs:
* ExpiringCacheObjectStore
* Cache samples.json
* Use Task Queues to avoid request timeout in the master server.
  Alternative: Use backends.
* Specify appropriate time for cron jobs.

Original comment by fangjue2...@gmail.com on 18 May 2013 at 11:29

GoogleCodeExporter commented 8 years ago 
ExpiringCacheObjectStore is done.
It turned out that individual requests to push update for a single file is 
better than one request for all files (otherwise cron is stuck), though I'm not 
sure why.

Since it's already running, lower the priority. There're many TODOs, though:
* Specify appropriate time for cron jobs (considering quotas available in a 
day). Make it automatic!!!
* Move API keys, etc. out and commit the source code.
* Write tests.
* Consider other optimizations.
* Polish StatUpdater (it's not correctly updated! look at how 
patched_file_system does it.)
* Investigate why cron is stuck?

Original comment by fangjue2...@gmail.com on 19 May 2013 at 1:59

GoogleCodeExporter commented 8 years ago 
oops. Cron jobs are super flaky. Is it due to too many db operations (or 
because of indexes?)? Maybe some classes (CompiledFileSystem, etc.) should use 
a non-persistent object stores.

Original comment by fangjue2...@gmail.com on 21 May 2013 at 11:08

GoogleCodeExporter commented 8 years ago 
Make ResponseObjectStore non-persistent (as well as other object stores used in 
server instance and compiled file system). Docs rendering was smooth but got 
stuck again in static/*.

Original comment by fangjue2...@gmail.com on 24 May 2013 at 11:30

GoogleCodeExporter commented 8 years ago 
Next step:
* Remove slave-samples.
* Add a task queue job to trigger the current cron job instead of running 
periodically or by manually update cron.yaml.

Original comment by fangjue2...@gmail.com on 1 Oct 2013 at 3:00

GoogleCodeExporter commented 8 years ago 
Mark as Fixed since everything is working now.

Original comment by fangjue2...@gmail.com on 28 Apr 2014 at 11:16