Endless Loading while creating feed

[JReming85]

After I select title & desc and click the next button

It is stuck on a endless spinning circle.

Logs show

"POST /setup_create_feed HTTP/1.0" 500 12604

in the frontend app

and

[twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fecd2a06aa0>, )

in the backend

Any ideas?

[taroved]

There is no enough information. Give me your page address

вт, 25 сент. 2018 г. в 5:59, Jeremy R notifications@github.com:

After I select title & desc and click the next button

It is stuck on a endless spinning circle.

Logs show

"POST /setup_create_feed HTTP/1.0" 500 12604

in the frontend app

and

[twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fecd2a06aa0>, )

in the backend

Any ideas?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/taroved/pol/issues/23, or mute the thread https://github.com/notifications/unsubscribe-auth/AFASWsAcjIj1dRBvRoUwEY8XP4SvBPlsks5ueZv4gaJpZM4W3x_W .

[JReming85]

Of course, it is http://pol.jreming.com

After trying to curl the "setup_create_feed"

It shows its failing due to Debug being set to True, but I have tried everything to get it to work turned off with no avail (3+ hours yesterday)

I finally expanded my ssl cert to include the new subdomain, but it still is failing.

Any ideas? I always get 500 server error and nothing loads with debug=False

[JReming85]

Further testing

I turned on logging and this is the result with Debug=False

[25/Sep/2018 15:25:24] ERROR [django.request:256] Internal Server Error: /en/ Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/pol/frontend/frontend/views.py", line 36, in index return render(request, 'frontend/index.html', {'form': form}) File "/usr/local/lib/python2.7/dist-packages/django/shortcuts.py", line 67, in render template_name, context, request=request, using=using) File "/usr/local/lib/python2.7/dist-packages/django/template/loader.py", line 99, in render_to_string return template.render(context, request) File "/usr/local/lib/python2.7/dist-packages/django/template/backends/django.py", line 74, in render return self.template.render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 210, in render return self._render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render return self.nodelist.render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 905, in render bit = self.render_node(node, context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 919, in render_node return node.render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/loader_tags.py", line 135, in render return compiled_parent._render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 202, in _render return self.nodelist.render(context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 905, in render bit = self.render_node(node, context) File "/usr/local/lib/python2.7/dist-packages/django/template/base.py", line 919, in render_node return node.render(context) File "/usr/local/lib/python2.7/dist-packages/pipeline/templatetags/pipeline.py", line 77, in render return self.render_compressed(package, 'css') File "/usr/local/lib/python2.7/dist-packages/pipeline/templatetags/pipeline.py", line 54, in render_compressed return method(package, package.output_filename) File "/usr/local/lib/python2.7/dist-packages/pipeline/templatetags/pipeline.py", line 84, in render_css 'url': mark_safe(staticfiles_storage.url(path)) File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/storage.py", line 131, in url hashed_name = self.stored_name(clean_name) File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/storage.py", line 280, in stored_name cache_name = self.clean_name(self.hashed_name(name)) File "/usr/local/lib/python2.7/dist-packages/django/contrib/staticfiles/storage.py", line 94, in hashed_name (clean_name, self)) ValueError: The file 'frontend/stylesheets/app.css' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x7f73aa48fed0>.

[taroved]

WIth Debug=False you shoud setup django assets compilation with command python manage.py collectstatic and add to nginx config something like

        location /static {
                alias /home/user/pol/frontend/frontend/static;
        }

[JReming85]

Looks like that did the trick,

Now I just need to figure out why the website I am trying to scrape's css files are getting blocked.

Thanks

[JReming85]

@taroved

Thanks for helping previously, but now I have another issue I would like to pick your brain about.

https://pol.jreming.com/feed/3?sanitize=Y is returning error 500

2018-09-25T12:37:58-0400 [twisted.python.log#info] "127.0.0.1" - - [25/Sep/2018:16:37:58 +0000] "GET /downloader?url=https%3A%2F%2Fwww.nbc12.com%2Fnews%2Fon-your-side%2F HTTP/1.0" 200 287175 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.61" 2018-09-25T12:37:58-0400 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fcb4c0fdcf8>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7fcb4b88bf10>) 2018-09-25T12:38:14-0400 [twisted.web.server.Request#critical] Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived why = self.lineReceived(line) File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 2099, in lineReceived self.allContentReceived() File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 2190, in allContentReceived req.requestReceived(command, path, version) File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 917, in requestReceived self.process() --- --- File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 198, in process self.render(resrc) File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 258, in render body = resrc.render(self) File "/usr/local/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render return m(request) File "/opt/pol/pol/server.py", line 345, in render_GET res = self.feed.getFeedData(feed_id) File "/opt/pol/pol/feed.py", line 160, in getFeedData where f.id=%s""", (feed_id,)) File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute self.errorhandler(self, exc, value) File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorvalue _mysql_exceptions.OperationalError: (1054, "Unknown column 'f.name' in 'field list'")

2018-09-25T12:38:14-0400 [twisted.python.log#info] "127.0.0.1" - - [25/Sep/2018:16:38:14 +0000] "GET /feed/3?sanitize=Y HTTP/1.0" 500 7060 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.61"

from download.py

/usr/local/lib/python2.7/dist-packages/twisted/web/server.py:198 in process 197 self._encoder = encoder 198 self.render(resrc) 199 except: /usr/local/lib/python2.7/dist-packages/twisted/web/server.py:258 in render 257 try: 258 body = resrc.render(self) 259 except UnsupportedMethod as e: /usr/local/lib/python2.7/dist-packages/twisted/web/resource.py:250 in render 249 raise UnsupportedMethod(allowedMethods) 250 return m(request) 251 /opt/pol/pol/server.py:345 in render_GET 344 else: 345 res = self.feed.getFeedData(feed_id) 346 /opt/pol/pol/feed.py:160 in getFeedData 159 left join frontend_field fi on fi.id=ff.field_id 160 where f.id=%s""", (feed_id,)) 161 rows = cur.fetchall() /usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py:226 in execute 225 exc, value = sys.exc_info()[:2] 226 self.errorhandler(self, exc, value) 227 self._executed = query /usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py:36 in defaulterrorhandler 35 if isinstance(errorvalue, BaseException): 36 raise errorvalue 37 if errorclass is not None: _mysql_exceptions.OperationalError: (1054, "Unknown column 'f.name' in 'field list'")

From the browser

Any Ideas?

[taroved]

The error says that column frontend_feeddoesn't have field name. Please check the field in the table with sql command desc frontend_feed Does it exists?

[taroved]

Seems the same issue as https://github.com/taroved/pol/issues/17

[JReming85]

Reinstalled SQL Parse, Dropped the database, recreated it, and reimported everything and it looks like I have come full circle.

From Browser:

Request URL: https://pol.jreming.com/setup_create_feed
Request Method: POST
Status Code: 500 INTERNAL SERVER ERROR

From Frontend:

Performing system checks...

System check identified no issues (0 silenced).
September 25, 2018 - 18:25:40
Django version 1.8.6, using settings 'frontend.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[25/Sep/2018 18:25:59] "GET / HTTP/1.0" 302 0
[25/Sep/2018 18:25:59] "GET /en/ HTTP/1.0" 200 5434
[25/Sep/2018 18:26:04] "GET /en/?url=https%3A%2F%2Fwww.nbc12.com%2Fnews%2Fon-your-side%2F HTTP/1.0" 302 0
[25/Sep/2018 18:26:05] "GET /en/setup?url=https%3A//www.nbc12.com/news/on-your-side/ HTTP/1.0" 200 12676
Fork path: /html/body/div/div/div/section/div/div/div/div/div/div/div/div/h4/a
title: 
[25/Sep/2018 18:26:12] "POST /setup_get_selected_ids HTTP/1.0" 200 466
Fork path: /html/body/div/div/div/section/div/div/div/div/div/div/div/div
description: div[1]
title: h4[1]/a[1]
[25/Sep/2018 18:26:14] "POST /setup_get_selected_ids HTTP/1.0" 200 823
[25/Sep/2018 18:26:16] "POST /setup_create_feed HTTP/1.0" 500 27

Fron Backend:

2018-09-25T14:25:51-0400 [-] Site starting on 1234
2018-09-25T14:25:51-0400 [twisted.web.server.Site#info] Starting factory <twisted.web.server.Site instance at 0x7f8d807026c8>
Request <GET https://www.nbc12.com/news/on-your-side/> started
2018-09-25T14:26:05-0400 [twisted.web.client._HTTP11ClientFactory#info] Starting factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f8d807006e0>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7f8d807145d0>)
Response <https://www.nbc12.com/news/on-your-side/> ready (20731 bytes)
2018-09-25T14:26:05-0400 [twisted.python.log#info] "127.0.0.1" - - [25/Sep/2018:18:26:05 +0000] "GET /downloader?url=https%3A%2F%2Fwww.nbc12.com%2Fnews%2Fon-your-side%2F HTTP/1.0" 200 289282 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.61"
2018-09-25T14:26:05-0400 [twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7f8d807006e0>, <twisted.internet.endpoints._WrapperEndpoint object at 0x7f8d807145d0>)

From Debug Log:

[25/Sep/2018 18:25:59] WARNING [django.request:170] Not Found: /
[25/Sep/2018 18:26:16] ERROR [django.request:256] Internal Server Error: /setup_create_feed
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/pol/frontend/frontend/views.py", line 129, in setup_create_feed
    feed_id = _create_feed(url, xpathes)
  File "/opt/pol/frontend/frontend/views.py", line 102, in _create_feed
    feed.save()
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
IntegrityError: (1364, "Field 'owner_id' doesn't have a default value")

Says owner_id doesnt have a default value, any idea what I am suppose to include?

[taroved]

Actually field owner_id of table frontend_feed has default value equal to NULL. But this exception is appears anyway. Remove default value and add it back. This should work And just questions for me: Which version of mysql do you use?

[JReming85]

I prefer MariaDB-server10 (normal) or PerconaDB (for a high rate of read/write), but on this server, one of my other applications required mysql5.7 so that's what it is running.

It looks like that solved it, had to do "owner_id", and then it complained about "name" so did that one too.

Thanks a lot for the help!

BTW, I noticed on the official site it has logins, is there any way to quickly implement that?

[taroved]

No. Login is not implemented for the open source version