search

Lessig2016 / pledgeservice

GNU Affero General Public License v3.0
9 stars 8 forks source link

New handler for exposing state-by-state donation totals. #8

Closed abliss closed 9 years ago

abliss commented 9 years ago

Moved from #4

furf commented 9 years ago

Thanks! I'll pull it down and test it now.

abliss commented 9 years ago

Thanks @aaronlifshin for inviting me to the trusted-tester site. I see from the logs there that the new endpoint is throwing 500 error due to

Property end_date must be a datetime, but was u'-'
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~lessig-trust-test/1.386876754693484313/handlers.py", line 750, in get
    for pledge in model.Pledge.all():
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2330, in next
    return self.__model_class.from_entity(self.__iterator.next())
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 1445, in from_entity
    return cls(None, _from_entity=entity, **entity_values)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 973, in __init__
    prop.__set__(self, value)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 617, in __set__
    value = self.validate(value)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 3018, in validate
    (self.name, self.data_type.__name__, value))
BadValueError: Property end_date must be a datetime, but was u'-'
furf commented 9 years ago

@abliss when i tried to run on GAE test instance, i hit two errors.

i resolved the first by using setdefault to initialize each state's total (https://github.com/Lessig2016/pledgeservice/compare/feature/state-by-state-totals?expand=1#diff-0d706bf2916b6f7a76178963618a8fdfR752).

the next is above my python pay grade. do you think you'll be able to take a look? we've added you to the GAE instance.

i see your comment above came in as i was typing. thoughts?

aaronlifshin commented 9 years ago

Yep... that is a strange error to get on a field that's empty across the DB.

On Wed, Sep 2, 2015 at 2:26 PM, Adam Bliss notifications@github.com wrote:

Thanks @aaronlifshin https://github.com/aaronlifshin for inviting me to the trusted-tester site. I see from the logs there that the new endpoint is throwing 500 error due to

Property end_date must be a datetime, but was u'-' Traceback (most recent call last): File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in call rv = self.handle_exception(request, response, e) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in call rv = self.router.dispatch(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher return route.handler_adapter(request, response) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in call return handler.dispatch() File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch return self.handle_exception(e, self.app.debug) File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch return method(_args, *_kwargs) File "/base/data/home/apps/s~lessig-trust-test/1.386876754693484313/handlers.py", line 750, in get for pledge in model.Pledge.all(): File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/init.py", line 2330, in next return self.model_class.from_entity(self.__iterator.next()) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/__init.py", line 1445, in from_entity return cls(None, _from_entity=entity, entity_values) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/init.py", line 973, in init prop.set(self, value) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/init.py", line 617, in set value = self.validate(value) File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/db/init.py", line 3018, in validate (self.name, self.data_type.name**, value)) BadValueError: Property end_date must be a datetime, but was u'-'

— Reply to this email directly or view it on GitHub https://github.com/Lessig2016/pledgeservice/pull/8#issuecomment-137249261 .

furf commented 9 years ago

all of the records returned by the Pledge.all query are missing the end_date value. possible to omit it from the specific query?

abliss commented 9 years ago

i think there is one record in the datastore which is messing up the query. I suggest deleting it or at least fixing its fields. id=5157979233976320

abliss commented 9 years ago

Oh, I see that it is a pledge by @aaronlifshin :)

Did you maybe enter that record manually (or while developing the end_date schema change)? It has '-' instead of a date field.

abliss commented 9 years ago

You can make the edit here :

https://appengine.google.com/datastore/edit?app_id=s~lessig-trust-test&version_id=1.386876754693484313&key=ahNzfmxlc3NpZy10cnVzdC10ZXN0chMLEgZQbGVkZ2UYgICAgPrklAkM

Probably just nulling out the "end_date" field will fix this error. (I shall abstain from monkeying with your datastore however :)

aaronlifshin commented 9 years ago

Edited that data.

On Wed, Sep 2, 2015 at 2:40 PM, Adam Bliss notifications@github.com wrote:

You can make the edit here :

https://appengine.google.com/datastore/edit?app_id=s~lessig-trust-test&version_id=1.386876754693484313&key=ahNzfmxlc3NpZy10cnVzdC10ZXN0chMLEgZQbGVkZ2UYgICAgPrklAkM

Probably just nulling out the "end_date" field will fix this error. (I shall abstain from monkeying with your datastore however :)

— Reply to this email directly or view it on GitHub https://github.com/Lessig2016/pledgeservice/pull/8#issuecomment-137252283 .

abliss commented 9 years ago

Similar issue with url_nonce on that record. (I don't know what an appropriate value would be, but the new 500 error is because it's empty.)

aaronlifshin commented 9 years ago

DELORTED

On Wed, Sep 2, 2015 at 2:49 PM, Adam Bliss notifications@github.com wrote:

Similar issue with url_nonce on that record.

— Reply to this email directly or view it on GitHub https://github.com/Lessig2016/pledgeservice/pull/8#issuecomment-137254240 .

abliss commented 9 years ago

Awesome, seems to work now!

abliss commented 9 years ago

So, any chance of seeing this pushed to production?

aaronlifshin commented 9 years ago

I am really concerned about the performance impact of this the way it is written.

Can we get it to run every 10 minutes or so through a queue, push the results to the datastore and have the endpoint just read that small datastore table?

A

On Thu, Sep 3, 2015 at 4:59 PM, Adam Bliss notifications@github.com wrote:

So, any chance of seeing this pushed to production?

— Reply to this email directly or view it on GitHub https://github.com/Lessig2016/pledgeservice/pull/8#issuecomment-137604635 .

aaronlifshin commented 9 years ago

Reading a large(ish) table like that in GAE is really not recommended, it's not the way the replication datastore is meant to be used.

On Thu, Sep 3, 2015 at 5:06 PM, Aaron Lifshin aaronlifshin@gmail.com wrote:

I am really concerned about the performance impact of this the way it is written.

Can we get it to run every 10 minutes or so through a queue, push the results to the datastore and have the endpoint just read that small datastore table?

A

On Thu, Sep 3, 2015 at 4:59 PM, Adam Bliss notifications@github.com wrote:

So, any chance of seeing this pushed to production?

— Reply to this email directly or view it on GitHub https://github.com/Lessig2016/pledgeservice/pull/8#issuecomment-137604635 .

abliss commented 9 years ago

How about https://github.com/Lessig2016/pledgeservice/pull/11 instead? (I don't have time to go learn the GAE queue api right now.)

Thanks for your time.