manishprajaptii commented 4 years ago

Hi Siddhartham, First of all thanks for your great work you did by creating this plugin to store images to digitalOcean. I am facing issue with saving images to digitalOceanSpaces. kindly look into it.

find my thumbor.config

################################### Logging ####################################

Logging configuration as json

Defaults to: None

THUMBOR_LOG_CONFIG = None

Log Format to be used by thumbor when writing log messages.

Defaults to: '%(asctime)s %(name)s:%(levelname)s %(message)s'

THUMBOR_LOG_FORMAT = '%(asctime)s %(name)s:%(levelname)s %(message)s'

Date Format to be used by thumbor when writing log messages.

Defaults to: '%Y-%m-%d %H:%M:%S'

THUMBOR_LOG_DATE_FORMAT = '%Y-%m-%d %H:%M:%S'

################################################################################

################################### Imaging ####################################

Max width in pixels for images read or generated by thumbor

Defaults to: 0

MAX_WIDTH = 0

Max height in pixels for images read or generated by thumbor

Defaults to: 0

MAX_HEIGHT = 0

Max pixel count for images read by thumbor

Defaults to: 75000000.0

MAX_PIXELS = 75000000.0

Min width in pixels for images read or generated by thumbor

Defaults to: 1

MIN_WIDTH = 1

Min width in pixels for images read or generated by thumbor

Defaults to: 1

MIN_HEIGHT = 1

Allowed domains for the http loader to download. These are regular

expressions.

Defaults to: [

]

ALLOWED_SOURCES = [

]

Quality index used for generated JPEG images

Defaults to: 80

QUALITY = 80

Exports JPEG images with the `progressive` flag set.

Defaults to: True

PROGRESSIVE_JPEG = True

Specify subsampling behavior for Pillow (see `subsampling` in

http://pillow.readthedocs.org/en/latest/handbook/image-file-

formats.html#jpeg).Be careful to use int for 0,1,2 and string for "4:4:4"

notation. Will ignore `quality`. Using `keep` will copy the original file's

subsampling.

Defaults to: None

PILLOW_JPEG_SUBSAMPLING = None

Specify quantization tables for Pillow (see `qtables` in

http://pillow.readthedocs.org/en/latest/handbook/image-file-

formats.html#jpeg). Will ignore `quality`. Using `keep` will copy the

original file's qtables.

Defaults to: None

PILLOW_JPEG_QTABLES = None

Specify resampling filter for Pillow resize method.One of LANCZOS, NEAREST,

BILINEAR, BICUBIC, HAMMING (Pillow>=3.4.0).

Defaults to: 'LANCZOS'

PILLOW_RESAMPLING_FILTER = 'LANCZOS'

Quality index used for generated WebP images. If not set (None) the same level

of JPEG quality will be used. If 100 the `lossless` flag will be used.

Defaults to: None

WEBP_QUALITY = None

Compression level for generated PNG images.

Defaults to: 6

PNG_COMPRESSION_LEVEL = 6

Indicates if final image should preserve indexed mode (P or 1) of original

image

Defaults to: True

PILLOW_PRESERVE_INDEXED_MODE = True

Specifies whether WebP format should be used automatically if the request

accepts it (via Accept header)

Defaults to: False

AUTO_WEBP = False

Specifies whether a PNG image should be used automatically if the png image

has no transparency (via alpha layer). WARNING: Depending on case, this is

not a good deal. This transformation maybe causes distortions or the size

of image can increase. Images with texts, for example, the result image

maybe will be distorced. Dark images, for example, the size of result image

maybe will be bigger. You have to evaluate the majority of your use cases

to take a decision about the usage of this conf.

Defaults to: False

AUTO_PNG_TO_JPG = False

Specify the ratio between 1in and 1px for SVG images. This is only used

whenrasterizing SVG images having their size units in cm or inches.

Defaults to: 150

SVG_DPI = 150

Max AGE sent as a header for the image served by thumbor in seconds

Defaults to: 86400

MAX_AGE = 86400

Indicates the Max AGE header in seconds for temporary images (images with

failed smart detection)

Defaults to: 0

MAX_AGE_TEMP_IMAGE = 0

Indicates whether thumbor should rotate images that have an Orientation EXIF

header

Defaults to: False

RESPECT_ORIENTATION = False

Ignore errors during smart detections and return image as a temp image (not

saved in result storage and with MAX_AGE_TEMP_IMAGE age)

Defaults to: False

IGNORE_SMART_ERRORS = False

Sends If-Modified-Since & Last-Modified headers; requires support from result

storage

Defaults to: False

SEND_IF_MODIFIED_LAST_MODIFIED_HEADERS = False

Preserves exif information in generated images. Increases image size in

kbytes, use with caution.

Defaults to: False

PRESERVE_EXIF_INFO = False

Indicates whether thumbor should enable the EXPERIMENTAL support for animated

gifs.

Defaults to: True

ALLOW_ANIMATED_GIFS = True

Indicates whether thumbor should use gifsicle engine. Please note that smart

cropping and filters are not supported for gifs using gifsicle (but won't

give an error).

Defaults to: False

USE_GIFSICLE_ENGINE = False

Indicates whether thumbor should enable blacklist functionality to prevent

processing certain images.

Defaults to: False

USE_BLACKLIST = False

Size of the thread pool used for image transformations. The default value is

0 (don't use a threadpoool. Increase this if you are seeing your IOLoop

getting blocked (often indicated by your upstream HTTP requests timing out)

Defaults to: 0

ENGINE_THREADPOOL_SIZE = 0

################################################################################

################################ Extensibility #################################

The metrics backend thumbor should use to measure internal actions. This must

be the full name of a python module (python must be able to import it)

Defaults to: 'thumbor.metrics.logger_metrics'

METRICS = 'thumbor.metrics.logger_metrics'

The loader thumbor should use to load the original image. This must be the

full name of a python module (python must be able to import it)

Defaults to: 'thumbor.loaders.http_loader'

LOADER = 'thumbor_spaces.loaders.spaces_loader'

The file storage thumbor should use to store original images. This must be the

full name of a python module (python must be able to import it)

Defaults to: 'thumbor.storages.file_storage'

STORAGE = 'thumbor.storages.file_storage'

The result storage thumbor should use to store generated images. This must be

the full name of a python module (python must be able to import it)

Defaults to: None

RESULT_STORAGE = 'thumbor_spaces.result_storages.spaces_storage'

The imaging engine thumbor should use to perform image operations. This must

be the full name of a python module (python must be able to import it)

Defaults to: 'thumbor.engines.pil'

ENGINE = 'thumbor.engines.pil'

The gif engine thumbor should use to perform image operations. This must be

the full name of a python module (python must be able to import it)

Defaults to: 'thumbor.engines.gif'

GIF_ENGINE = 'thumbor.engines.gif'

The url signer thumbor should use to verify url signatures.This must be the

full name of a python module (python must be able to import it)

Defaults to: 'libthumbor.url_signers.base64_hmac_sha1'

URL_SIGNER = 'libthumbor.url_signers.base64_hmac_sha1'

################################################################################

################################### Security ###################################

The security key thumbor uses to sign image URLs

Defaults to: 'MY_SECURE_KEY'

SECURITY_KEY = 'MY_SECURE_KEY'

Indicates if the /unsafe URL should be available

Defaults to: True

ALLOW_UNSAFE_URL = True

Indicates if encrypted (old style) URLs should be allowed

Defaults to: True

ALLOW_OLD_URLS = True

################################################################################

##################################### HTTP #####################################

Enables automatically generated etags

Defaults to: True

ENABLE_ETAGS = True

################################################################################

################################### Storage ####################################

Set maximum id length for images when stored

Defaults to: 32

MAX_ID_LENGTH = 32

################################################################################

################################# Performance ##################################

Set garbage collection interval in seconds

Defaults to: None

GC_INTERVAL = None

################################################################################

################################# Healthcheck ##################################

Healthcheck route.

Defaults to: '/healthcheck'

HEALTHCHECK_ROUTE = '/healthcheck'

################################################################################

################################### Metrics ####################################

Host to send statsd instrumentation to

Defaults to: None

STATSD_HOST = None

Port to send statsd instrumentation to

Defaults to: 8125

STATSD_PORT = 8125

Prefix for statsd

Defaults to: None

STATSD_PREFIX = None

################################################################################

################################# File Loader ##################################

The root path where the File Loader will try to find images

Defaults to: '/home/java-manish'

FILE_LOADER_ROOT_PATH = '/home/java-manish'

################################################################################

################################# HTTP Loader ##################################

The maximum number of seconds libcurl can take to connect to an image being

loaded

Defaults to: 5

HTTP_LOADER_CONNECT_TIMEOUT = 5

The maximum number of seconds libcurl can take to download an image

Defaults to: 20

HTTP_LOADER_REQUEST_TIMEOUT = 20

Indicates whether libcurl should follow redirects when downloading an image

Defaults to: True

HTTP_LOADER_FOLLOW_REDIRECTS = True

Indicates the number of redirects libcurl should follow when downloading an

image

Defaults to: 5

HTTP_LOADER_MAX_REDIRECTS = 5

The maximum number of simultaneous HTTP connections the loader can make before

queuing

Defaults to: 10

HTTP_LOADER_MAX_CLIENTS = 10

Indicates whether thumbor should forward the user agent of the requesting user

Defaults to: False

HTTP_LOADER_FORWARD_USER_AGENT = False

Indicates whether thumbor should forward the headers of the request

Defaults to: False

HTTP_LOADER_FORWARD_ALL_HEADERS = False

Indicates which headers should be forwarded among all the headers of the

request

Defaults to: [

]

HTTP_LOADER_FORWARD_HEADERS_WHITELIST = [

]

Default user agent for thumbor http loader requests

Defaults to: 'Thumbor/6.7.1'

HTTP_LOADER_DEFAULT_USER_AGENT = 'Thumbor/6.7.1'

The proxy host needed to load images through

Defaults to: None

HTTP_LOADER_PROXY_HOST = None

The proxy port for the proxy host

Defaults to: None

HTTP_LOADER_PROXY_PORT = None

The proxy username for the proxy host

Defaults to: None

HTTP_LOADER_PROXY_USERNAME = None

The proxy password for the proxy host

Defaults to: None

HTTP_LOADER_PROXY_PASSWORD = None

The filename of CA certificates in PEM format

Defaults to: None

HTTP_LOADER_CA_CERTS = None

Validate the server’s certificate for HTTPS requests

Defaults to: None

HTTP_LOADER_VALIDATE_CERTS = None

The filename for client SSL key

Defaults to: None

HTTP_LOADER_CLIENT_KEY = None

The filename for client SSL certificate

Defaults to: None

HTTP_LOADER_CLIENT_CERT = None

If the CurlAsyncHTTPClient should be used

Defaults to: False

HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT = False

################################################################################

################################### General ####################################

If HTTP_LOADER_CURL_LOW_SPEED_LIMIT and HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT are

set, then this is the time in seconds as integer after a download should

timeout if the speed is below HTTP_LOADER_CURL_LOW_SPEED_LIMIT for that

long

Defaults to: 0

HTTP_LOADER_CURL_LOW_SPEED_TIME = 0

If HTTP_LOADER_CURL_LOW_SPEED_TIME and HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT are

set, then this is the limit in bytes per second as integer which should

timeout if the speed is below that limit for

HTTP_LOADER_CURL_LOW_SPEED_TIME seconds

Defaults to: 0

HTTP_LOADER_CURL_LOW_SPEED_LIMIT = 0

Custom app class to override ThumborServiceApp. This config value is

overridden by the -a command-line parameter.

Defaults to: 'thumbor.app.ThumborServiceApp'

APP_CLASS = 'thumbor.app.ThumborServiceApp'

################################################################################

################################# File Storage #################################

Expiration in seconds for the images in the File Storage. Defaults to one

month

Defaults to: 2592000

STORAGE_EXPIRATION_SECONDS = 2592000

Indicates whether thumbor should store the signing key for each image in the

file storage. This allows the key to be changed and old images to still be

properly found

Defaults to: False

STORES_CRYPTO_KEY_FOR_EACH_IMAGE = False

The root path where the File Storage will try to find images

Defaults to: '/tmp/thumbor/storage'

FILE_STORAGE_ROOT_PATH = '/tmp/thumbor/storage'

################################################################################

#################################### Upload ####################################

Max size in Kb for images uploaded to thumbor

Aliases: MAX_SIZE

Defaults to: 0

UPLOAD_MAX_SIZE = 0

Indicates whether thumbor should enable File uploads

Aliases: ENABLE_ORIGINAL_PHOTO_UPLOAD

Defaults to: False

UPLOAD_ENABLED = True

The type of storage to store uploaded images with

Aliases: ORIGINAL_PHOTO_STORAGE

Defaults to: 'thumbor.storages.file_storage'

UPLOAD_PHOTO_STORAGE = 'thumbor_spaces.result_storages.spaces_storage'

Indicates whether image deletion should be allowed

Aliases: ALLOW_ORIGINAL_PHOTO_DELETION

Defaults to: False

UPLOAD_DELETE_ALLOWED = True

Indicates whether image overwrite should be allowed

Aliases: ALLOW_ORIGINAL_PHOTO_PUTTING

Defaults to: False

UPLOAD_PUT_ALLOWED = True

Default filename for image uploaded

Defaults to: 'image'

UPLOAD_DEFAULT_FILENAME = 'image'

################################################################################

############################### Memcache Storage ###############################

List of Memcache storage server hosts

Defaults to: [

'localhost:11211',

]

MEMCACHE_STORAGE_SERVERS = [

'localhost:11211',

]

################################################################################

################################ Mixed Storage #################################

Mixed Storage file storage. This must be the full name of a python module

(python must be able to import it)

Defaults to: 'thumbor.storages.no_storage'

MIXED_STORAGE_FILE_STORAGE = 'thumbor.storages.no_storage'

Mixed Storage signing key storage. This must be the full name of a python

module (python must be able to import it)

Defaults to: 'thumbor.storages.no_storage'

MIXED_STORAGE_CRYPTO_STORAGE = 'thumbor.storages.no_storage'

Mixed Storage detector information storage. This must be the full name of a

python module (python must be able to import it)

Defaults to: 'thumbor.storages.no_storage'

MIXED_STORAGE_DETECTOR_STORAGE = 'thumbor.storages.no_storage'

################################################################################

##################################### Meta #####################################

The callback function name that should be used by the META route for JSONP

access

Defaults to: None

META_CALLBACK_NAME = None

################################################################################

################################## Detection ###################################

List of detectors that thumbor should use to find faces and/or features. All

of them must be full names of python modules (python must be able to import

it)

Defaults to: [

]

DETECTORS = [

]

The cascade file that opencv will use to detect faces.

Defaults to: 'haarcascade_frontalface_alt.xml'

FACE_DETECTOR_CASCADE_FILE = 'haarcascade_frontalface_alt.xml'

The cascade file that opencv will use to detect glasses.

Defaults to: 'haarcascade_eye_tree_eyeglasses.xml'

GLASSES_DETECTOR_CASCADE_FILE = 'haarcascade_eye_tree_eyeglasses.xml'

The cascade file that opencv will use to detect profile faces.

Defaults to: 'haarcascade_profileface.xml'

PROFILE_DETECTOR_CASCADE_FILE = 'haarcascade_profileface.xml'

################################################################################

################################## Optimizers ##################################

List of optimizers that thumbor will use to optimize images

Defaults to: [

]

OPTIMIZERS = [

]

Path for the jpegtran binary

Defaults to: '/usr/bin/jpegtran'

JPEGTRAN_PATH = '/usr/bin/jpegtran'

Path for the progressive scans file to use with jpegtran optimizer. Implies

progressive jpeg output

Defaults to: ''

JPEGTRAN_SCANS_FILE = ''

Path for the ffmpeg binary used to generate gifv(h.264)

Defaults to: '/usr/local/bin/ffmpeg'

FFMPEG_PATH = '/usr/local/bin/ffmpeg'

################################################################################

################################### Filters ####################################

List of filters that thumbor will allow to be used in generated images. All of

them must be full names of python modules (python must be able to import

it)

Defaults to: [

'thumbor.filters.brightness',

'thumbor.filters.colorize',

'thumbor.filters.contrast',

'thumbor.filters.rgb',

'thumbor.filters.round_corner',

'thumbor.filters.quality',

'thumbor.filters.noise',

'thumbor.filters.watermark',

'thumbor.filters.equalize',

'thumbor.filters.fill',

'thumbor.filters.sharpen',

'thumbor.filters.strip_exif',

'thumbor.filters.strip_icc',

'thumbor.filters.frame',

'thumbor.filters.grayscale',

'thumbor.filters.rotate',

'thumbor.filters.format',

'thumbor.filters.max_bytes',

'thumbor.filters.convolution',

'thumbor.filters.blur',

'thumbor.filters.extract_focal',

'thumbor.filters.focal',

'thumbor.filters.no_upscale',

'thumbor.filters.saturation',

'thumbor.filters.max_age',

'thumbor.filters.curve',

'thumbor.filters.background_color',

'thumbor.filters.upscale',

'thumbor.filters.proportion',

'thumbor.filters.stretch',

]

FILTERS = [

'thumbor.filters.brightness',

'thumbor.filters.colorize',

'thumbor.filters.contrast',

'thumbor.filters.rgb',

'thumbor.filters.round_corner',

'thumbor.filters.quality',

'thumbor.filters.noise',

'thumbor.filters.watermark',

'thumbor.filters.equalize',

'thumbor.filters.fill',

'thumbor.filters.sharpen',

'thumbor.filters.strip_exif',

'thumbor.filters.strip_icc',

'thumbor.filters.frame',

'thumbor.filters.grayscale',

'thumbor.filters.rotate',

'thumbor.filters.format',

'thumbor.filters.max_bytes',

'thumbor.filters.convolution',

'thumbor.filters.blur',

'thumbor.filters.extract_focal',

'thumbor.filters.focal',

'thumbor.filters.no_upscale',

'thumbor.filters.saturation',

'thumbor.filters.max_age',

'thumbor.filters.curve',

'thumbor.filters.background_color',

'thumbor.filters.upscale',

'thumbor.filters.proportion',

'thumbor.filters.stretch',

]

################################################################################

################################ Result Storage ################################

Expiration in seconds of generated images in the result storage

Defaults to: 0

RESULT_STORAGE_EXPIRATION_SECONDS = 0

Path where the Result storage will store generated images

Defaults to: '/tmp/thumbor/result_storage'

RESULT_STORAGE_FILE_STORAGE_ROOT_PATH = '/tmp/thumbor/result_storage'

Indicates whether unsafe requests should also be stored in the Result Storage

Defaults to: False

RESULT_STORAGE_STORES_UNSAFE = True

################################################################################

############################ Queued Redis Detector #############################

Server host for the queued redis detector

Defaults to: 'localhost'

REDIS_QUEUE_SERVER_HOST = 'localhost'

Server port for the queued redis detector

Defaults to: 6379

REDIS_QUEUE_SERVER_PORT = 6379

Server database index for the queued redis detector

Defaults to: 0

REDIS_QUEUE_SERVER_DB = 0

Server password for the queued redis detector

Defaults to: None

REDIS_QUEUE_SERVER_PASSWORD = None

################################################################################

############################# Queued SQS Detector ##############################

AWS key id

Defaults to: None

SQS_QUEUE_KEY_ID = None

AWS key secret

Defaults to: None

SQS_QUEUE_KEY_SECRET = None

AWS SQS region

Defaults to: 'us-east-1'

SQS_QUEUE_REGION = 'us-east-1'

################################################################################

#################################### Errors ####################################

This configuration indicates whether thumbor should use a custom error

handler.

Defaults to: False

USE_CUSTOM_ERROR_HANDLING = False

Error reporting module. Needs to contain a class called ErrorHandler with a

handle_error(context, handler, exception) method.

Defaults to: 'thumbor.error_handlers.sentry'

ERROR_HANDLER_MODULE = 'thumbor.error_handlers.sentry'

File of error log as json

Defaults to: None

ERROR_FILE_LOGGER = None

File of error log name is parametrized with context attribute

Defaults to: False

ERROR_FILE_NAME_USE_CONTEXT = False

################################################################################

############################### Errors - Sentry ################################

Sentry thumbor project dsn. i.e.: http://5a63d58ae7b94f1dab3dee740b301d6a:73ee

a45d3e8649239a973087e8f21f98@localhost:9000/2

Defaults to: ''

SENTRY_DSN_URL = ''

Sentry environment i.e.: staging

Defaults to: None

SENTRY_ENVIRONMENT = None

################################################################################

#################################### Server ####################################

The amount of time to wait before shutting down the server, i.e. stop

accepting requests.

Defaults to: 0

MAX_WAIT_SECONDS_BEFORE_SERVER_SHUTDOWN = 0

The amount of time to waut before shutting down all io, after the server has

been stopped

Defaults to: 0

MAX_WAIT_SECONDS_BEFORE_IO_SHUTDOWN = 0

################################################################################

############################### Digital Ocean #################################

configuration for digital ocean

region

SPACES_REGION='stage-fnp'

SPACE ENDPOINT

SPACES_ENDPOINT='stage-fnp.sfo2'

SPACES_KEY

SPACES_KEY='xxxx'

SPACES_SECRET

SPACES_SECRET='xxxxxxx'

SPACES_BUCKET

SPACES_BUCKET='stage-fnp'

SPACES_ACL

SPACES_ACL='public'

SPACES_RESULT_FOLDER

SPACES_RESULT_FOLDER='result_storage'

SPACES_LOADER_FOLDER

SPACES_LOADER_FOLDER='storage'

ghost commented 4 years ago

Hello. @manishprajaptii Could you check your thumbor log and let me know what is the error? Thanks

denzfarid commented 3 years ago

hellow again :) im have same problem, my thumbor error says

curl -I localhost:8000/unsafe/0x0:2000x1020/600x800/staging/media/orig_50ada34821434780a99b734ad2ab4b79.jpg

HTTP/1.1 504 Gateway Timeout
Date: Mon, 07 Sep 2020 12:22:20 GMT
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Server: TornadoServer/5.1.1

2020-09-07 19:22:20 thumbor:WARNING ERROR retrieving image http://staging/media/orig_50ada34821434780a99b734ad2ab4b79.jpg: [Errno -2] Name or service not known

2020-09-07 19:22:20 tornado.access:ERROR 504 HEAD /unsafe/0x0:2000x1020/600x800/staging/media/orig_50ada34821434780a99b734ad2ab4b79.jpg (10.130.64.23) 11.82ms

maybe thumbor retrive url http://staging/media/orig_50ada34821434780a99b734ad2ab4b79.jpg not url digital ocean space

denzfarid commented 3 years ago

Update again resolve :) after I read some of the code,

https://github.com/int2root/thumbor_spaces/blob/14e1aa123b93b40272895a82a6d3b6e233b02989/thumbor_spaces/loaders/spaces_loader.py#L22

and

https://github.com/int2root/thumbor_spaces/blob/14e1aa123b93b40272895a82a6d3b6e233b02989/thumbor_spaces/loaders/spaces_loader.py#L23

I just understood that the SPACES_ENDPOINT variable was only in the region and not the full url, because the code has been set to default value .digitaloceanspaces.com, if we enter the full endpoint what will happen is, sgp1.digitaloceanspaces.com.digitaloceanspaces.com

and the SPACES_LOADER_FOLDER variable is filled by the folder we want to go, in the example it says SPACES_LOADER_FOLDER = 'storage' that means there is a storage folder in space, in my case, I don't have a storage folder but have a staging folder, so I replace storage with staging

like this is my current configuration thumbor.conf,

##################
# Digital Ocean Space  #
##################

RESULT_STORAGE = 'thumbor_spaces.result_storages.spaces_storage'
LOADER = 'thumbor_spaces.loaders.spaces_loader'
SPACES_REGION='sgp1'
SPACES_ENDPOINT='sgp1'
SPACES_KEY='******'
SPACES_SECRET='******'
SPACES_BUCKET='my-bucket'
SPACES_ACL='public'
SPACES_RESULT_FOLDER='result_storage'
SPACES_LOADER_FOLDER='staging'

hope it can help

highoncarbs commented 3 years ago

@denzfarid @int2root I have set up thumbor , but the result storage doesn't save anything to DO spaces. Am I missing something ? Here's my config

RESULT_STORAGE = 'thumbor_spaces.result_storages.spaces_storage'
LOADER = 'thumbor_spaces.loaders.spaces_loader'

SPACES_REGION='ams3'

SPACES_ENDPOINT='ams3'

SPACES_KEY='***'

SPACES_SECRET='***'

SPACES_BUCKET='saas-space' #name of spaces instance

SPACES_ACL='public'

SPACES_RESULT_FOLDER='result_storage'

SPACES_LOADER_FOLDER='panya' # name of folder in my bucket

Help is much appreciated. Thanks

denzfarid commented 3 years ago

@denzfarid @int2root I have set up thumbor , but the result storage doesn't save anything to DO spaces. Am I missing something ? Here's my config
RESULT_STORAGE = 'thumbor_spaces.result_storages.spaces_storage'
LOADER = 'thumbor_spaces.loaders.spaces_loader'

SPACES_REGION='ams3'

SPACES_ENDPOINT='ams3'

SPACES_KEY='***'

SPACES_SECRET='***'

SPACES_BUCKET='saas-space' #name of spaces instance

SPACES_ACL='public'

SPACES_RESULT_FOLDER='result_storage'

SPACES_LOADER_FOLDER='panya' # name of folder in my bucket
Help is much appreciated. Thanks

please check your thumbor error sir,

btw now I don't use this module because I have a problem with the image name, I have a problem with whitespace (url encode) if I have an image with spaces, this module renders an image with%20, so the image is always not found

example: I have an image with the name: thumbor space.jpg each space is encoded by the browser to the thumbor%20space.jpg and this module still served with the thumbor%20space.jpg, even though the image is called thumbor space.jpg, not thumbor% 20space.jpg.

currently I use s3fs and I mount it like nfs, so that the thumbor space.jpg file will still be called as the file name thumbor space.jpg, not as thumbor% 20space.jpg

highoncarbs commented 3 years ago

Hi , I have no errors at the moment , everything thing runs. But I suppose the result storage should store it in DO spaces right ? Somehow it's not happening. P.S : You can always replace the HTML escape chars

highoncarbs commented 3 years ago

@denzfarid Also what I meant is that thumbor should save the result file after cropping and resizing back to spaces itself and not to my droplet , is this possible ?

denzfarid commented 3 years ago

@denzfarid Also what I meant is that thumbor should save the result file after cropping and resizing back to spaces itself and not to my droplet , is this possible ?

@highoncarbs do you mean cache? at this time I don't save the cropped results and also the smart face detection results in the file, instead I save it in redis, and I also save the image in redis, so that the cost of traffic on the space is not too large, because the saved image is no longer taken from bukcet space

https://thumbor.readthedocs.io/en/latest/lazy_detection.html

https://github.com/thumbor-community/redis

highoncarbs commented 3 years ago

No, I meant the result storage / cache of the images that have been already processed . Someone from thumbor community said that the aws plugin should also work the same as the spaces plugin , so I'm gonna go with that

denzfarid commented 3 years ago

No, I meant the result storage / cache of the images that have been already processed . Someone from thumbor community said that the aws plugin should also work the same as the spaces plugin , so I'm gonna go with that

actually, I had that problem too :D

highoncarbs commented 3 years ago

@denzfarid Any ideas on fixing it , it seems that aws plugin also has the same issue. What would you recommend ?

denzfarid commented 3 years ago

@denzfarid Any ideas on fixing it , it seems that aws plugin also has the same issue. What would you recommend ?

im already mention it sir currently I use s3fs and I mount it like nfs

https://github.com/s3fs-fuse/s3fs-fuse

s3fs can mount , aliyun, digital ocean, ... etc maybe all cloud storage with S3-compatible

this example blog for mounting space with s3fs https://medium.com/simplebackups/mount-digitalocean-spaces-bucket-on-linux-cee6a64db4b9

highoncarbs commented 3 years ago

@denzfarid Thanks a lot , Didn't really get it at first . Really appreciate your help

highoncarbs commented 3 years ago

@denzfarid So sorry to bug you again . I have mounted my DO Spaces bucket , I can view all the files and manually add the files like - touch ok.txt , but Thumbor doesn't seem to write and save the images in my bucket , here's my thumbor config -

RESULT_STORAGE = 'thumbor.result_storages.file_storage'
LOADER = 'thumbor_spaces.loaders.spaces_loader'

SPACES_REGION='ams3'

SPACES_ENDPOINT='ams3'

SPACES_KEY='xxxxx'

SPACES_SECRET='xxxxx'

SPACES_BUCKET='saas-space' #name of spaces instance

SPACES_ACL='public'

SPACES_RESULT_FOLDER='result_storage'

RESULT_STORAGE_FILE_STORAGE_ROOT_PATH = '/media/panya-result'

SPACES_LOADER_FOLDER='panya' # name of folder in my bucket

s3fs Debug Logs

Nov  6 10:16:00 panya s3fs[6263]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 10:16:00 panya s3fs[6263]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 10:16:01 panya s3fs[6263]: s3fs.cpp:s3fs_check_service(3813): bucket not found(host=https://ams3.digitaloceanspaces.com) - result of checking service.
Nov  6 10:19:15 panya s3fs[6305]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 10:19:15 panya s3fs[6305]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 16:01:47 panya s3fs[7585]: s3fs.cpp:set_s3fs_log_level(257): change debug level from [CRT] to [DBG]
Nov  6 16:01:47 panya s3fs[7585]:     PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40700)
Nov  6 16:02:15 panya s3fs[7589]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 16:02:15 panya s3fs[7589]: check services.
Nov  6 16:02:15 panya s3fs[7589]:       check a bucket.
Nov  6 16:02:15 panya s3fs[7589]: curl.cpp:GetHandler(283): Get handler from pool: 31
Nov  6 16:02:15 panya s3fs[7589]:       URL is https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:15 panya s3fs[7589]:       URL changed is https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:15 panya s3fs[7589]:       computing signature [GET] [/] [] []
Nov  6 16:02:15 panya s3fs[7589]:       url is https://ams3.digitaloceanspaces.com
Nov  6 16:02:15 panya s3fs[7589]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 16:02:15 panya s3fs[7589]: curl.cpp:RequestPerform(1923): connecting to URL https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:16 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:20 panya s3fs[7589]: ### retrying...
Nov  6 16:02:20 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:20 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:24 panya s3fs[7589]: ### retrying...
Nov  6 16:02:24 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:25 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:29 panya s3fs[7589]: ### retrying...
Nov  6 16:02:29 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:RequestPerform(2089): ### giving up
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:CheckBucket(2953): Check bucket failed, S3 response:
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_check_service(3820): unable to connect(host=https://ams3.digitaloceanspaces.com) - result of checking service.
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:ReturnHandler(306): Return handler to pool: 31
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_exit_fuseloop(3368): Exiting FUSE event loop due to errors
Nov  6 16:02:29 panya s3fs[7589]: destroy
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_destroy(3445): Could not release curl library.

S3Fs start command

 s3fs saas-space /media/panya-result -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel="debug" -o url=https://ams3.digitaloceanspaces.com/ -o use_path_request_style

Would really appreciate if you could guide me around this. Thanks a lot for all the help again , though

denzfarid commented 3 years ago

@denzfarid So sorry to bug you again . I have mounted my DO Spaces bucket , I can view all the files and manually add the files like - touch ok.txt , but Thumbor doesn't seem to write and save the images in my bucket , here's my thumbor config -

RESULT_STORAGE = 'thumbor.result_storages.file_storage'
LOADER = 'thumbor_spaces.loaders.spaces_loader'

SPACES_REGION='ams3'

SPACES_ENDPOINT='ams3'

SPACES_KEY='xxxxx'

SPACES_SECRET='xxxxx'

SPACES_BUCKET='saas-space' #name of spaces instance

SPACES_ACL='public'

SPACES_RESULT_FOLDER='result_storage'

RESULT_STORAGE_FILE_STORAGE_ROOT_PATH = '/media/panya-result'

SPACES_LOADER_FOLDER='panya' # name of folder in my bucket

s3fs Debug Logs

Nov  6 10:16:00 panya s3fs[6263]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 10:16:00 panya s3fs[6263]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 10:16:01 panya s3fs[6263]: s3fs.cpp:s3fs_check_service(3813): bucket not found(host=https://ams3.digitaloceanspaces.com) - result of checking service.
Nov  6 10:19:15 panya s3fs[6305]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 10:19:15 panya s3fs[6305]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 16:01:47 panya s3fs[7585]: s3fs.cpp:set_s3fs_log_level(257): change debug level from [CRT] to [DBG]
Nov  6 16:01:47 panya s3fs[7585]:     PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40700)
Nov  6 16:02:15 panya s3fs[7589]: s3fs.cpp:s3fs_init(3378): init v1.82(commit:unknown) with GnuTLS(gcrypt)
Nov  6 16:02:15 panya s3fs[7589]: check services.
Nov  6 16:02:15 panya s3fs[7589]:       check a bucket.
Nov  6 16:02:15 panya s3fs[7589]: curl.cpp:GetHandler(283): Get handler from pool: 31
Nov  6 16:02:15 panya s3fs[7589]:       URL is https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:15 panya s3fs[7589]:       URL changed is https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:15 panya s3fs[7589]:       computing signature [GET] [/] [] []
Nov  6 16:02:15 panya s3fs[7589]:       url is https://ams3.digitaloceanspaces.com
Nov  6 16:02:15 panya s3fs[7589]: Libgcrypt warning: missing initialization - please fix the application
Nov  6 16:02:15 panya s3fs[7589]: curl.cpp:RequestPerform(1923): connecting to URL https://ams3.digitaloceanspaces.com/saas-space/
Nov  6 16:02:16 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:20 panya s3fs[7589]: ### retrying...
Nov  6 16:02:20 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:20 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:24 panya s3fs[7589]: ### retrying...
Nov  6 16:02:24 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:25 panya s3fs[7589]:       HTTP response code 503
Nov  6 16:02:29 panya s3fs[7589]: ### retrying...
Nov  6 16:02:29 panya s3fs[7589]:       Retry request. [type=5][url=https://ams3.digitaloceanspaces.com/saas-space/][path=/]
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:RequestPerform(2089): ### giving up
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:CheckBucket(2953): Check bucket failed, S3 response:
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_check_service(3820): unable to connect(host=https://ams3.digitaloceanspaces.com) - result of checking service.
Nov  6 16:02:29 panya s3fs[7589]: curl.cpp:ReturnHandler(306): Return handler to pool: 31
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_exit_fuseloop(3368): Exiting FUSE event loop due to errors
Nov  6 16:02:29 panya s3fs[7589]: destroy
Nov  6 16:02:29 panya s3fs[7589]: s3fs.cpp:s3fs_destroy(3445): Could not release curl library.

S3Fs start command

 s3fs saas-space /media/panya-result -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel="debug" -o url=https://ams3.digitaloceanspaces.com/ -o use_path_request_style

Would really appreciate if you could guide me around this. Thanks a lot for all the help again , though

okay after it is mounted, then you don't need to use this plugin anymore, you should use the thumbor configuration with the disk filesystem as before, because it is assumed, the file is already on the OS we are using

https://github.com/denzfarid/config/tree/master/webserver/thumbor

highoncarbs commented 3 years ago

@denzfarid Yes ! Thanks again mate. You are a lifesaver

denzfarid commented 3 years ago

@denzfarid Yes ! Thanks again mate. You are a lifesaver

this my thumbor config loader

LOADER = 'thumbor.loaders.file_loader'
STORAGE = 'thumbor.storages.file_storage'
FILE_LOADER_ROOT_PATH = '/mnt/space'

/mnt/space

is location im mounting space with s3fs

and this is my /etc/fstab config

Space-bucket  /mnt/space fuse.s3fs _netdev,allow_other,use_path_request_style,url=https://sgp1.digitaloceanspaces.com 0 0

mount in /etc/fstab for help autostart after rebooting server (upgrade spec or whatever)

highoncarbs commented 3 years ago

@denzfarid Thanks for the tip man. Also one question , did you have any issues with thumbor filters not working on cloud , but working on your local machine ?

denzfarid commented 3 years ago

@denzfarid Thanks for the tip man. Also one question , did you have any issues with thumbor filters not working on cloud , but working on your local machine ?

no, everything is fine, maybe some php extensions and packages are not installed

highoncarbs commented 3 years ago

@denzfarid Thanks for everything, really! I ended up using thumbor_aws plugin , s3fs fuse took a lot of toll on my server memory.

mukherjee78 / thumbor_spaces

Not Connecting with DigitialOcean #2

Logging configuration as json

Defaults to: None

THUMBOR_LOG_CONFIG = None

Log Format to be used by thumbor when writing log messages.

Defaults to: '%(asctime)s %(name)s:%(levelname)s %(message)s'

THUMBOR_LOG_FORMAT = '%(asctime)s %(name)s:%(levelname)s %(message)s'

Date Format to be used by thumbor when writing log messages.

Defaults to: '%Y-%m-%d %H:%M:%S'

THUMBOR_LOG_DATE_FORMAT = '%Y-%m-%d %H:%M:%S'

Max width in pixels for images read or generated by thumbor

Defaults to: 0

MAX_WIDTH = 0

Max height in pixels for images read or generated by thumbor

Defaults to: 0

MAX_HEIGHT = 0

Max pixel count for images read by thumbor

Defaults to: 75000000.0

MAX_PIXELS = 75000000.0

Min width in pixels for images read or generated by thumbor

Defaults to: 1

MIN_WIDTH = 1

Min width in pixels for images read or generated by thumbor

Defaults to: 1

MIN_HEIGHT = 1

Allowed domains for the http loader to download. These are regular

expressions.

Defaults to: [

]

ALLOWED_SOURCES = [

]

Quality index used for generated JPEG images

Defaults to: 80

QUALITY = 80

Exports JPEG images with the progressive flag set.

Defaults to: True

PROGRESSIVE_JPEG = True

Specify subsampling behavior for Pillow (see subsampling in

http://pillow.readthedocs.org/en/latest/handbook/image-file-

formats.html#jpeg).Be careful to use int for 0,1,2 and string for "4:4:4"

notation. Will ignore quality. Using keep will copy the original file's

subsampling.

Defaults to: None

PILLOW_JPEG_SUBSAMPLING = None

Specify quantization tables for Pillow (see qtables in

http://pillow.readthedocs.org/en/latest/handbook/image-file-

formats.html#jpeg). Will ignore quality. Using keep will copy the

original file's qtables.

Defaults to: None

PILLOW_JPEG_QTABLES = None

Specify resampling filter for Pillow resize method.One of LANCZOS, NEAREST,

BILINEAR, BICUBIC, HAMMING (Pillow>=3.4.0).

Defaults to: 'LANCZOS'

PILLOW_RESAMPLING_FILTER = 'LANCZOS'

Quality index used for generated WebP images. If not set (None) the same level

of JPEG quality will be used. If 100 the lossless flag will be used.

Defaults to: None

WEBP_QUALITY = None

Compression level for generated PNG images.

Defaults to: 6

PNG_COMPRESSION_LEVEL = 6

Indicates if final image should preserve indexed mode (P or 1) of original

image

Defaults to: True

PILLOW_PRESERVE_INDEXED_MODE = True

Specifies whether WebP format should be used automatically if the request

accepts it (via Accept header)

Defaults to: False

AUTO_WEBP = False

Specifies whether a PNG image should be used automatically if the png image

has no transparency (via alpha layer). WARNING: Depending on case, this is

not a good deal. This transformation maybe causes distortions or the size

of image can increase. Images with texts, for example, the result image

maybe will be distorced. Dark images, for example, the size of result image

maybe will be bigger. You have to evaluate the majority of your use cases

to take a decision about the usage of this conf.

Defaults to: False

AUTO_PNG_TO_JPG = False

Specify the ratio between 1in and 1px for SVG images. This is only used

Exports JPEG images with the `progressive` flag set.

Specify subsampling behavior for Pillow (see `subsampling` in

notation. Will ignore `quality`. Using `keep` will copy the original file's

Specify quantization tables for Pillow (see `qtables` in

formats.html#jpeg). Will ignore `quality`. Using `keep` will copy the

of JPEG quality will be used. If 100 the `lossless` flag will be used.