search

a4k-openproject / script.module.openscrapers

OpenScrapers Project
GNU General Public License v3.0
102 stars 40 forks source link

Updated cache module and source_utils #101

Closed sraedler closed 4 years ago

sraedler commented 4 years ago

Changed caching method to pickle dump and added additional parameters. This enables the module to cache any type of data. This is especially helpful to use the cache module for cfscrape requests.

In the source_utils, a function is added to create a distinct search array from title, localtitle and aliases.

sraedler commented 4 years ago

This project is in hopes to unify the community and contribute to one scraper pack for multi scraper add-ons and not having the repo go dead or disappear.

Yeah, this Pull Request show, that the project is also not how the communities are going to come together and work on one base. If no maintainer even reply after more than a week, than it's not motivating to contribute.

drinfernoo commented 4 years ago

@sraedler Considering that this is a relatively small open source project with few contributors and maintainers to have gotten the ball rolling and keep it that way, and that, to my knowledge, nobody that has had any involvement in this project has made it their full time job yet... it's not unreasonable to assume that after a week something might go unlooked at.

The main Kodi project in GitHub has issues and pull requests that have been open for weeks or months, for goodness sakes.

reddit-reaper commented 4 years ago

@sraedler sorry about that i never got the email about this PR so never checked. Will look into also there's going to be some branch conflicts as we had updated somethings these last few days

sraedler commented 4 years ago

Sorry for being not respectful enough to you. I just thought you saw it and don‘t care about external contributions.

I would recommend to use the issue tracker, because otherwise things are done twice and there will never be more developers...

And my last cent: I would love, if you could use the develop branch as its name. Because contributions are than more easy. I would really like to contribute from time to time...

If you have questions about the PR, please ask!

sraedler commented 4 years ago

@reddit-reaper should I resolve the conflict?

reddit-reaper commented 4 years ago

Better yet, message me on Reddit. Reddit_reaper.

nazegnl commented 4 years ago

@reddit-reaper and @sraedler We need to be careful by using pickle, because the py2.7 pickle is alot slower and should nog be used. If we can use cPickle this is alot faster on py2. In py3 the pickle import uses cPickle if enabled in the python distribution.

I'm not sure if all supported platforms support cPickle, we need testing to confirm this will work on Android for example.

sraedler commented 4 years ago

@reddit-reaper I sent you a message on reddit @nazegnl You are right. Since Pickle is available on Android, a try catch import might be an option

nazegnl commented 4 years ago

@sraedler I think regular pickle for py2 is out of the question. We need cPickle else performance will be bad.

sraedler commented 4 years ago

@nazegnl I ment something like 

try: import cPickle as pickle
except: import Pickle as pickle