search

valeriansaliou / bloom

:cherry_blossom: HTTP REST API caching middleware, to be used between load balancers and REST API workers.
https://crates.io/crates/bloom-server
Mozilla Public License 2.0
712 stars 48 forks source link

Unable to workaround acquired empty meta value from cache #25

Open TheMlok opened 1 year ago

TheMlok commented 1 year ago

Hello there,

I like your idea very much. I would love to use bloom server as a cache for my backend JSON REST API written in Rust. But I am unable to make it work. I spent an hours trying, but no idea. I have prepared my environment for testing on Debian Linux (Buster).

I have installed bloom server using cargo install bloom-server, now I have bloom-server 1.35.2 installed (output from bloom -V).

My redis version is 7.0.11-1rl1~buster1. Configured accordingly to your example config (also used for another caching, so I use different database number, as you will se later).

This is my bloom config:

# Bloom
# HTTP REST API caching middleware
# Configuration file
# Example: https://github.com/valeriansaliou/bloom/blob/master/config.cfg

[server]
log_level = "info"
inet = "127.0.0.1:8051"

[control]
inet = "127.0.0.1:8081"
tcp_timeout = 300

[proxy]
shard_default = 0

[[proxy.shard]]
shard = 0
host = "localhost"
port = 8205

[cache]
ttl_default = 60
executor_pool = 64

disable_read = false
disable_write = false

compress_body = true

[redis]
host = "localhost"
port = 6379

database = 41

pool_size = 80
max_lifetime_seconds = 60
idle_timeout_seconds = 600
connection_timeout_seconds = 1

max_key_size = 256000
max_key_expiration = 2592000

My nginx is proxy passing this way:

 proxy_pass http://127.0.0.1:8051;
        proxy_set_header Bloom-Request-Shard 0;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

Before there are also CORS setups (also carefully modified by your docs) but my issue is not here, because I can get my data.

Problem is, that I am missing the cache. Header Bloom-Status is MISS still. I tried to add the headers from my API Bloom-Response-TTL or without. Everything is cached into redis (keys are visible) but then I got an error. This is Bloom log (INFO level)

INFO) - handled request: GET on /api/v2/articles
(INFO) - tunneling for ns = bloom:0:c:dc56d17a:c0b944a5
(INFO) - acquired empty meta value from cache
(INFO) - handled request: GET on /api/v2/notices
(INFO) - tunneling for ns = bloom:0:c:dc56d17a:dd2cc6f4
(INFO) - acquired empty meta value from cache

In my redis database I can see:

127.0.0.1:6379[41]> keys *
1) "bloom:0:a:dc56d17a"

etc. these logs are from history not in the same moment. I am trying to figure it out.

Please have you any suggestion? I would like to have Bloom working for me.

Thanks

valeriansaliou commented 1 year ago

The metadata key looks like this one: bloom:0:c:90d52bc6:f773d6f1, so it's definitely not stored in your Bloom instance, indicating that cache is not there / was never written.

Can you please share the raw HTTP response of the request that you'd like cached / which doesnt get cached, from your API? I'd need full HTTP response w/ headers and body.

TheMlok commented 1 year ago

Hello, yes of course. Sorry for late recall, I was busy on the other things and I need to simulate it on the server. Here is my response headers (body is just plain JSON)

* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=admin.mixit.ro.mix-it.cloud
*  start date: Oct 22 14:47:29 2023 GMT
*  expire date: Jan 20 14:47:28 2024 GMT
*  subjectAltName: host "admin.mixit.ro.mix-it.cloud" matched cert's "admin.mixit.ro.mix-it.cloud"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: admin.mixit.ro.mix-it.cloud]
* h2 [:path: /api/v2/news]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15c80fc00)
> GET /api/v2/news HTTP/2
> Host: admin.mixit.ro.mix-it.cloud
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 200
< server: nginx/1.22.0
< date: Mon, 23 Oct 2023 05:32:45 GMT
< content-type: application/json
< content-length: 11401
< vary: Accept-Encoding
< x-content-type-options: nosniff
< permissions-policy: interest-cohort=()
< x-frame-options: SAMEORIGIN
< vary: ETag
< etag: "33dbe95101d86171"
< bloom-status: MISS
<
{"news":[{"hasPadding":false,"id":256,"image":{"lg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_lg-e6d3555628e2855c4290e0443719be84.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_lg-9d4d69447b39d10edc6d40d6c0ff0762.webp"},"height":930,"width":1240},"llg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_llg-2c06a1874cd206ef41ad459447348fb4.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_llg-e88b92dcc324f40a5644ee31ee46b656.webp"},"height":1158,"width":1544},"lmd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_lmd-d2ef45cfda5b10a0f89a9c26e12422d5.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_lmd-5bc1d1063cb71cac14405c1582dcb5f0.webp"},"height":465,"width":620},"md":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_md-fa20f94ecdd2f78e9fe549ea2f8ffe7f.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_md-2f8f732847f67e325b157c2e6527810e.webp"},"height":437,"width":582},"mlg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_mlg-4d231a2a0eff0f3ad7b96c10550fd588.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_mlg-6d5d0202795f50bb22201274d3c4b930.webp"},"height":579,"width":772},"slg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_slg-d327e4cdf9d6b9453ef7c0f97b6a9cfd.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_slg-8713529d7e99fc0109f13c58b8ae4c5b.webp"},"height":495,"width":660},"sm":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_sm-5d9b93897fbce4a716ec717464be320e.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_sm-bd6a8af8ffa040de8b65a677c1a2b51f.webp"},"height":202,"width":269},"smd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_smd-dfcf6bad5e45ce6ac370ada5e71c5ff5.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_smd-012e590d65bdee0420d90eb67035becf.webp"},"height":264,"width":352},"xs":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/256/jpeg_xs-c5551731d94b64c1db86dcf12061e91c.jpeg","image/webp":"https://img.mix.it/news/ro/256/webp_xs-7e27308f0e0a674226d66efcc60ebcb8.webp"},"height":190,"width":253}},"link":"https://www.mixit.ro/product/terci-de-craciun-2","text":"Așteptăm Crăciunul cu voie bună și burtica plină cu terci de turtă dulce! Ne încălzește și ne scurtează așteptarea! ","title":"Noul terci de Turtă Dulce"},{"hasPadding":false,"id":255,"image":{"lg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_lg-4496eb49dd2a51b831711171e651d434.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_lg-93a1e4a14e859783aed4ba02b13ac993.webp"},"height":930,"width":1240},"llg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_llg-abe4b7163286d0bb3f547a9fc8612f98.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_llg-db65afe897f4c7a4016ece19ddae9a72.webp"},"height":1158,"width":1544},"lmd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_lmd-8b3ecf71563933f238e32df8c8f43780.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_lmd-f0ad42cf8f08ae1dadf73d9d1332b850.webp"},"height":465,"width":620},"md":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_md-c5850975edfd04614018fc3190b4469e.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_md-a8601bc0b61f41a577e7bb3e68079604.webp"},"height":437,"width":582},"mlg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_mlg-5da7817e42433d1e10a7dcfadc33933c.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_mlg-39c1491e1ab5cd1c6899ab3f1fa0c914.webp"},"height":579,"width":772},"slg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_slg-aad2c17d481b1d80c56cbd516d5ae2ac.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_slg-b23e81474f1a0c1cba75ae3b61b9fc67.webp"},"height":495,"width":660},"sm":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_sm-f5365cadc92a85be59170ebd24c35ef9.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_sm-4b9714ba8e0803cb12fb3a08e6cd316c.webp"},"height":202,"width":269},"smd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_smd-859772789ff0a5ae3022fc739832ec1e.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_smd-973336d1d0db5c5223937a24f9c2260c.webp"},"height":264,"width":352},"xs":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/255/jpeg_xs-6278eb704f5d98a538daad06a17bc567.jpg","image/webp":"https://img.mix.it/news/ro/255/webp_xs-da53ba9a27c629efa46298cde2ab69d4.webp"},"height":190,"width":253}},"link":"https://www.mixit.ro/mixit-pregatit/cadouri","text":"Cauți cadouri, care chiar vor bucura? Cu Seturile cadou Mixit mergi la sigur. ","title":"Set Cadouri"},{"hasPadding":false,"id":254,"image":{"lg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_lg-8ff1fcdcfd0c5b5fe79755d71d1b151a.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_lg-10e74cab5b4afd7e2ac3c5024ec6208e.webp"},"height":930,"width":1240},"llg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_llg-16282606732c6f4714eb66809b077fcf.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_llg-b023c2f1ddb9a88d23f1f8b1d5e947fd.webp"},"height":1158,"width":1544},"lmd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_lmd-d774fe71c302910ee1a9debd04fc3722.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_lmd-b7d972fad2b62dcb39f1b0759989119e.webp"},"height":465,"width":620},"md":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_md-950654c91fa7bfba0bde301ad83594ca.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_md-d5a4782ce2f1350c2e61f33289c520b3.webp"},"height":437,"width":582},"mlg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_mlg-3db054ea46d0c83f4eb2233e228e1842.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_mlg-63d37304444bbb2b891cd260e6b60c77.webp"},"height":579,"width":772},"slg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_slg-0921fa488e5c0c6e682fc6aa1929fb8c.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_slg-8bb33a3c0af30a14bec45b6c62c2db38.webp"},"height":495,"width":660},"sm":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_sm-3ea3c27285a456c6498c86c4c0e10d86.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_sm-49b7f4ade106ca354bfb04638c0c0a7e.webp"},"height":202,"width":269},"smd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_smd-07979f8d62651f59fa2b32db048ecda6.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_smd-7786dd97e7b60ec6a15d328bb8f61d06.webp"},"height":264,"width":352},"xs":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/254/jpeg_xs-5c4c0f2774047bca7def4d7d48b1da8e.jpg","image/webp":"https://img.mix.it/news/ro/254/webp_xs-4534b5e820f7fec4f4f7445c98b31cd6.webp"},"height":190,"width":253}},"link":"https://www.mixit.ro/product/biluta-cremoasa-cinnamon-chocolate","text":"Colinde vor răsuna când vei gusta aceste bomboane cu gustul Crăciunului! Răsfață-te de acum cu gusturi de sărbătoare. ","title":"Gustul Crăciunului"},{"hasPadding":false,"id":251,"image":{"lg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_lg-205c801cbfa170faa75132bcb6acf8fc.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_lg-6bc003600af4fccdd886e621ed245679.webp"},"height":930,"width":1240},"llg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_llg-00c4ba10552efa8f4a2eb0df2b5f7deb.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_llg-8421acc5479cd28dfb8ee186780d1b9e.webp"},"height":1158,"width":1544},"lmd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_lmd-073828a4dd4eebcb6f006ecb5d5efdfa.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_lmd-5f3dbc89b6729ab4634e606ada492cd8.webp"},"height":465,"width":620},"md":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_md-872531e1b3811f03f275065292e54fd8.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_md-848a80004db846f3e3af8f70947a2983.webp"},"height":437,"width":582},"mlg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_mlg-2a7ed5aa7b5f67c329fc2922b9a731ca.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_mlg-3e2f56da800a96fd7b5bcdddb8e0a90a.webp"},"height":579,"width":772},"slg":{"formats":{"image/jpeg":"https://img.mix.it* Connection #0 to host admin.mixit.ro.mix-it.cloud left intact
/news/ro/251/jpeg_slg-663b7e829f88f7458261d5d71d561e5e.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_slg-9a7b4b7aa161585defb83a3af4a4baab.webp"},"height":495,"width":660},"sm":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_sm-2f0593196f905f980947e88a3e40a5f4.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_sm-d1cb16eba38adb562758d04caf99235a.webp"},"height":202,"width":269},"smd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_smd-a5650d109a88e5fa4aea3801d948f353.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_smd-f39285187a01fd0c3a3eb6bee6b7b7b6.webp"},"height":264,"width":352},"xs":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/251/jpeg_xs-de85433bcb7dea297d863ac8bc19fca3.jpg","image/webp":"https://img.mix.it/news/ro/251/webp_xs-35e8c09518a27f31c7503f9637c578a8.webp"},"height":190,"width":253}},"link":"https://www.mixit.ro/product/mix-de-craciun-2-2","text":"Pregătește-te din timp și savurează magia Crăciunului un timp cât mai lung. Noi am început deja, pentru că este cea mai fermecată perioadă a anului! ","title":"Intră în spiritul Crăciunului!"},{"hasPadding":false,"id":252,"image":{"lg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_lg-6318cffff9a5fce3797ba95da3b9ca91.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_lg-71286958d0d796f02a53d7a33f2892d2.webp"},"height":930,"width":1240},"lmd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_lmd-5cc9d3a5b7da017c456af81198d52395.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_lmd-bd52c5f6e3b1b3ea7b0b8659c0876ac2.webp"},"height":465,"width":620},"md":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_md-e849c368298f061b047dc0e503ca5dff.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_md-f3b5d793e0f81ab9b8be30a0e7926dcf.webp"},"height":437,"width":582},"mlg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_mlg-75b893ab06d77101099b1da0bd2b6660.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_mlg-e7a9bb954348c429b015061f02f1530b.webp"},"height":579,"width":772},"slg":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_slg-3ef4576498acf7409f49ddc9f85be08b.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_slg-fb869f3f3ac02ab809ffe4d548487692.webp"},"height":495,"width":660},"sm":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_sm-e99aba5a078729449bec6047a195d04c.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_sm-4ab6bf6f19b681969daadab4fbdfa119.webp"},"height":202,"width":269},"smd":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_smd-0b7743665c15d85f633fba55fccfea6a.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_smd-30271dbb599e2f1d394e81b329d57820.webp"},"height":264,"width":352},"xs":{"formats":{"image/jpeg":"https://img.mix.it/news/ro/252/jpeg_xs-354ff3280716dc21ad2fe8347e0ac52c.jpg","image/webp":"https://img.mix.it/news/ro/252/webp_xs-bc117ef99935eba33e83e8d5f63e5f66.webp"},"height":190,"width":253}},"link":"https://www.mixit.ro/product/marele-calendar-mixit-advent-2023","text":"Comandă cele 24+2 de ferestre pline cu surprize și noutăți delicioase. ","title":"Marele calendar Mixit - Advent 2023"}]}

Follows just raw json data.

This is my Bloom log from first request on this URL, second follows with headers I pasted here.

(INFO) - starting up
(INFO) - binding to store backend at localhost:6379
(INFO) - bound to store backend
(INFO) - listening on tcp://127.0.0.1:8081
(INFO) - listening on http://127.0.0.1:8051
(INFO) - handled request: GET on /api/v2/news
(INFO) - tunneling for ns = bloom:0:c:dc56d17a:2a3ac6d2
(INFO) - acquired empty meta value from cache
(INFO) - handled request: GET on /api/v2/news
(INFO) - tunneling for ns = bloom:0:c:dc56d17a:2a3ac6d2
(INFO) - acquired empty meta value from cache

Thanks for the help.

valeriansaliou commented 1 year ago

I don’t see any Bloom header on your request. Did you set them somewhere? It needs to be set at the reverse proxy level so that Bloom knows which upstream to route to and what to do.

TheMlok commented 1 year ago

Yes, my location in nginx has proxy_set_header Bloom-Request-Shard 0 set on the routes.

What I posted you is response from server, and you can see bloom-status: MISS. I also try to set headers in my API (code below) When I go to route where is caching turned off it sets proper bloom-status to DIRECT.

This is my fairing for Rocket:


#[rocket::async_trait]
impl Fairing for CustomHeadersFairing {
    fn info(&self) -> Info {
        Info {
            name: "Custom Bloom caching Headers Fairing",
            kind: Kind::Response,
        }
    }

    async fn on_response<'r>(&self, _req: &'r Request<'_>, res: &mut Response<'r>) {
        // Convert the request path to a string and conditionally set headers
        let path = _req.uri().path().to_string();
        match path {
            ref s if s == "/api/v2/mix_categories" => {
                res.set_raw_header("Bloom-Response-TTL", "600");
            }
            ref s if s == "/api/v2/product_categories" => {
                res.set_raw_header("Bloom-Response-TTL", "600");
            }
            ref s if s == "/api/v2/notices" => {
                res.set_raw_header("Bloom-Response-Ignore", "1");
            }
            _ => {
                res.set_raw_header("Bloom-Response-TTL", &CONFIG.default_cache_ttl);
            }
        }
    }
}

And this is my CORS settings:

        proxy_cache off;
        if ($http_origin = "https://app.mixit.ro.mix-it.cloud") {
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' '*' always;
            add_header 'Access-Control-Expose-Headers' 'Vary, ETag' always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
        }

        # Handle preflight requests (OPTIONS)
        if ($request_method = 'OPTIONS') {
            add_header 'Access-Control-Allow-Origin' "$http_origin" always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' '*' always;
            add_header 'Access-Control-Expose-Headers' 'Vary, ETag' always;
            add_header 'Access-Control-Allow-Credentials' 'true' always;
            add_header 'Access-Control-Max-Age' 1728000;
            add_header 'Content-Type' 'application/json charset=UTF-8';
            add_header 'Content-Length' 0;
            return 204;
        }

If I hit notices I get DIRECT as status, so I think processing works, I had also HIT once but I was unable to repeat it, I don't see what I am getting wrong :(.

EDIT: removes unnecessary comments