search

sonatype-nexus-community / nexus-repository-composer

Composer support for the Nexus Repository Manager (work in progress!)
Eclipse Public License 1.0
216 stars 83 forks source link

In the new version of the plugin (Java 17) the composer-proxy is cached incorrectly #160

Open 3v01ut10n opened 5 days ago

3v01ut10n commented 5 days ago

composer-proxy is configured in the Nexus v3.71+. The latest version of the plugin nexus-repository-composer is used.

On cached packages, when rebuild one and the same php project, the package is not taken from the cache, but is constantly downloaded.

The blob store size keeps growing until it reaches the disk maximum. After that, only re-creating the blob store helps until the size reaches the limit again.

Nexus logs:

2024-09-17 06:09:49,107+0000 WARN  [qtp1105945070-111] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@74e3ec24, contentType='application/json'}, attributes='{etag=f668fa612254d48b1cfacd6f79bef10c0c391a7a, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2588, path='/packages.json', kind='PACKAGES', componentId=null, component=null, assetBlobId=514911, assetBlob=AssetBlobData{assetBlobId=514911, blobRef=nexus-composer-proxy@a201e816-9885-4d4d-9fb1-655c59add54c, blobSize=16879739, contentType='application/json', checksums={sha256=f2664b0cfc354779e5f0fd860f5e6ce3244be1efce4682939acbdc93d324e76f, md5=546df86edbc6fd7ae0efa7aa6b8fd3e3, sha1=f668fa612254d48b1cfacd6f79bef10c0c391a7a}, blobCreated=2024-09-17T05:49:11.178Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:10:06.099775Z, assetSize=16879739} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:10:06.099658Z, lastUpdated=2024-09-17T05:49:11.179250Z}, last_modified=2024-09-17T05:49:11.178Z}'}, assuming stale content.
2024-09-17 06:09:49,271+0000 WARN  [qtp1105945070-111] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@6112abbd, contentType='application/json'}, attributes='{etag=2cce33dd5c3f1a283590be74ca4d63d6458bc387, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2587, path='/packages/list.json', kind='LIST', componentId=null, component=null, assetBlobId=514910, assetBlob=AssetBlobData{assetBlobId=514910, blobRef=nexus-composer-proxy@ff87d460-6925-43f0-909c-6d24ac15d56f, blobSize=10868847, contentType='application/json', checksums={sha256=7754bbb1c30a2f4a76785e20bbbf61933409453519d716ac614fb24e8913bb0b, md5=510ae489143b81ef1a54c586868759c7, sha1=2cce33dd5c3f1a283590be74ca4d63d6458bc387}, blobCreated=2024-09-17T05:49:10.383Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:10:05.109180Z, assetSize=10868847} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:10:05.109063Z, lastUpdated=2024-09-17T05:49:10.385168Z}, last_modified=2024-09-17T05:49:10.383Z}'}, assuming stale content.

For correct caching of packages (composer).

Check the cache and cache retrieval function when requesting a package.

stklcode commented 2 days ago

Does it only affect the packages.json and packages/list.json form the quoted warning? If so, I probably got an idea what is missing here. Will likely look at it over the weekend.

3v01ut10n commented 2 days ago

I checked the latest logs. The problem occurs not only with packages.json and packages/list.json. But with a variety of packages:

2024-09-20 06:11:48,103+0000 WARN  [qtp1105945070-7298] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@583cbbb4, contentType='application/zip'}, attributes='{etag=06e470f45012ed077a1ea9da129bc73c7f44f1f6, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=4053, path='/sebastian/type/5.1.0/sebastian-type-5.1.0.zip', kind='ZIPBALL', componentId=2836, component=ComponentData{componentId=2836, namespace='sebastian', name='type', kind='', version='5.1.0', normalizedVersion='000000005.000000001.000000000', entityVersion='null'} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing={name=sebastian/type, description=Collection of value objects that represent the types of the PHP type system, type=library, homepage=https://github.com/sebastianbergmann/type, license=BSD-3-Clause, authors=[Sebastian Bergmann <sebastian@phpunit.de>], support_issues=https://github.com/sebastianbergmann/type/issues}}, created=2024-09-19T20:05:39.893042Z, lastUpdated=2024-09-19T20:05:39.897319Z}, assetBlobId=577943, assetBlob=AssetBlobData{assetBlobId=577943, blobRef=nexus-composer-proxy@c520a79b-eb70-4393-9741-11a6e1a8bdf6, blobSize=23693, contentType='application/zip', checksums={sha1=06e470f45012ed077a1ea9da129bc73c7f44f1f6, md5=e73f92f7b1ed498901800e9193f17984, sha256=038d416c36a32d5eeda0d21f1ff74bbfaa480c07030388b58073b9276e2d4eec}, blobCreated=2024-09-20T06:04:04.210Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-19T20:05:39.899595Z, assetSize=23693} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-19T20:05:39.899518Z, lastUpdated=2024-09-20T06:04:04.211927Z}, last_modified=2024-09-20T06:04:04.210Z}'}, assuming stale content.
2024-09-20 06:11:48,103+0000 WARN  [qtp1105945070-7306] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@46c7d612, contentType='application/zip'}, attributes='{etag=90e6d1f22eee80dec8c07d362eb13ca2990f6897, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=4055, path='/phpunit/phpunit/11.3.6/phpunit-phpunit-11.3.6.zip', kind='ZIPBALL', componentId=2838, component=ComponentData{componentId=2838, namespace='phpunit', name='phpunit', kind='', version='11.3.6', normalizedVersion='000000011.000000003.000000006', entityVersion='null'} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing={name=phpunit/phpunit, description=The PHP Unit Testing framework., type=library, keywords=[phpunit, xunit, testing], homepage=https://phpunit.de/, license=BSD-3-Clause, authors=[Sebastian Bergmann <sebastian@phpunit.de>], support_issues=https://github.com/sebastianbergmann/phpunit/issues}}, created=2024-09-19T20:05:40.091944Z, lastUpdated=2024-09-19T20:05:40.095084Z}, assetBlobId=577946, assetBlob=AssetBlobData{assetBlobId=577946, blobRef=nexus-composer-proxy@d0119f19-cfab-47a7-bf85-1b94b9b48636, blobSize=948051, contentType='application/zip', checksums={sha1=90e6d1f22eee80dec8c07d362eb13ca2990f6897, md5=041fdf8dcabf29985a25828233caaeed, sha256=69a54021cf428eb434d483d61a2a3186a9a04d788e50b7eb0cd26c3739a0a803}, blobCreated=2024-09-20T06:04:05.509Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-19T20:05:40.097200Z, assetSize=948051} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-19T20:05:40.097147Z, lastUpdated=2024-09-20T06:04:05.510767Z}, last_modified=2024-09-20T06:04:05.509Z}'}, assuming stale content.
2024-09-20 06:11:48,104+0000 WARN  [qtp1105945070-7298] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@435d117, contentType='application/json'}, attributes='{etag=22b97f0644d7e1355c568307ad244df6b758721e, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2795, path='/p2/sebastian/type.json', kind='PACKAGE', componentId=null, component=null, assetBlobId=577939, assetBlob=AssetBlobData{assetBlobId=577939, blobRef=nexus-composer-proxy@04fcaf89-e629-448d-a7f9-15e2b448ed55, blobSize=15619, contentType='application/json', checksums={sha1=22b97f0644d7e1355c568307ad244df6b758721e, md5=742a4fe5786384735734c7be94c4ae9d, sha256=2c0c7e1d630d267cfae4d4693a18304832de788bf8ac5999b62b35e9215603dd}, blobCreated=2024-09-20T06:04:04.032Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:17:44.131402Z, assetSize=15619} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:17:44.131336Z, lastUpdated=2024-09-20T06:04:04.034304Z}, last_modified=2024-09-20T06:04:04.032Z}'}, assuming stale content.
2024-09-20 06:11:48,105+0000 WARN  [qtp1105945070-7306] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@62e5ae16, contentType='application/json'}, attributes='{etag=ab6e34f620ca8a2fe0fdd0913e6e8beb032a7844, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2845, path='/p2/phpunit/phpunit.json', kind='PACKAGE', componentId=null, component=null, assetBlobId=577942, assetBlob=AssetBlobData{assetBlobId=577942, blobRef=nexus-composer-proxy@222a4e3b-e538-485b-84e8-e2d0cae8810a, blobSize=448071, contentType='application/json', checksums={sha1=ab6e34f620ca8a2fe0fdd0913e6e8beb032a7844, md5=a41d53100351df8f6734bd212e9f8005, sha256=1618b0af578e5823de27aa705c029b8ec550deddd1c6131ddd48ffbaba14b2b1}, blobCreated=2024-09-20T06:04:04.126Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:17:44.979346Z, assetSize=448071} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:17:44.979275Z, lastUpdated=2024-09-20T06:04:04.127861Z}, last_modified=2024-09-20T06:04:04.126Z}'}, assuming stale content.
2024-09-20 06:11:48,105+0000 WARN  [qtp1105945070-7288] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@15748d91, contentType='application/zip'}, attributes='{etag=3daf7b6af005d2e8c2e77586e2d6eef6e2b3560a, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=4056, path='/phpstan/phpstan/1.12.4/phpstan-phpstan-1.12.4.zip', kind='ZIPBALL', componentId=2839, component=ComponentData{componentId=2839, namespace='phpstan', name='phpstan', kind='', version='1.12.4', normalizedVersion='000000001.000000012.000000004', entityVersion='null'} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing={name=phpstan/phpstan, description=PHPStan - PHP Static Analysis Tool, keywords=[dev, static analysis], license=[MIT], support_issues=https://github.com/phpstan/phpstan/issues, support_forum=https://github.com/phpstan/phpstan/discussions, support_source=https://github.com/phpstan/phpstan-src, support_docs=https://phpstan.org/user-guide/getting-started}}, created=2024-09-19T20:05:43.462740Z, lastUpdated=2024-09-19T20:05:43.465836Z}, assetBlobId=577945, assetBlob=AssetBlobData{assetBlobId=577945, blobRef=nexus-composer-proxy@ed1e6f5f-a83f-4dea-9f7d-3f71719d708a, blobSize=3643228, contentType='application/zip', checksums={sha1=3daf7b6af005d2e8c2e77586e2d6eef6e2b3560a, md5=b51e8886fd09ffd4f11624b4cbdb0b4f, sha256=382ab4a68ced42534f37d6b7cc75eadc8573728715de7fbd6978b8bd3a1b3830}, blobCreated=2024-09-20T06:04:04.630Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-19T20:05:43.467874Z, assetSize=3643228} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-19T20:05:43.467810Z, lastUpdated=2024-09-20T06:04:04.632220Z}, last_modified=2024-09-20T06:04:04.630Z}'}, assuming stale content.
2024-09-20 06:11:48,106+0000 WARN  [qtp1105945070-7283] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@12f913a3, contentType='application/zip'}, attributes='{etag=595f51e4d38c6072e38d80b52be39ab6acf1214a, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=4054, path='/nikic/php-parser/v5.2.0/nikic-php-parser-v5.2.0.zip', kind='ZIPBALL', componentId=2837, component=ComponentData{componentId=2837, namespace='nikic', name='php-parser', kind='', version='v5.2.0', normalizedVersion='v000000005.000000002.000000000', entityVersion='null'} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing={name=nikic/php-parser, description=A PHP parser written in PHP, type=library, keywords=[php, parser], license=BSD-3-Clause, authors=[Nikita Popov]}}, created=2024-09-19T20:05:40.083439Z, lastUpdated=2024-09-19T20:05:40.086118Z}, assetBlobId=577944, assetBlob=AssetBlobData{assetBlobId=577944, blobRef=nexus-composer-proxy@9cfdf78e-e55d-4826-8770-21e41dde15d5, blobSize=269113, contentType='application/zip', checksums={sha1=595f51e4d38c6072e38d80b52be39ab6acf1214a, md5=0dde98b1f24f4720368a832b8393a129, sha256=f244d93ed144a308bffc1ae711deb80875f128348052556b4f30da22c8a21f80}, blobCreated=2024-09-20T06:04:04.223Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-19T20:05:40.088279Z, assetSize=269113} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-19T20:05:40.088192Z, lastUpdated=2024-09-20T06:04:04.224416Z}, last_modified=2024-09-20T06:04:04.223Z}'}, assuming stale content.
2024-09-20 06:11:48,107+0000 WARN  [qtp1105945070-7283] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@20abc75, contentType='application/json'}, attributes='{etag=865d7ba317ba3073bc1dcc5d8f2b37c094b299cb, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2826, path='/p2/nikic/php-parser.json', kind='PACKAGE', componentId=null, component=null, assetBlobId=577940, assetBlob=AssetBlobData{assetBlobId=577940, blobRef=nexus-composer-proxy@7e59f483-7e0e-47cb-b7d1-5d1d6b31f88b, blobSize=53598, contentType='application/json', checksums={sha1=865d7ba317ba3073bc1dcc5d8f2b37c094b299cb, md5=436577b3d4f5c345cc953788f2bf9ae6, sha256=53e39b1b1bc73eebea7fab1997b3e97d1dbd4b6f7c12d6de967dc369169ce8a5}, blobCreated=2024-09-20T06:04:04.033Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:17:44.705283Z, assetSize=53598} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:17:44.705216Z, lastUpdated=2024-09-20T06:04:04.034550Z}, last_modified=2024-09-20T06:04:04.033Z}'}, assuming stale content.
2024-09-20 06:11:48,108+0000 WARN  [qtp1105945070-7288] *UNKNOWN org.sonatype.nexus.repository.composer.internal.proxy.ComposerProxyFacet - CacheInfo missing for Content{payload=BlobPayload{blob=org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob@1ea79fcf, contentType='application/json'}, attributes='{etag=aa994748be8d221e2dba828360d45cb80045b3e6, org.sonatype.nexus.repository.content.Asset=AssetData{assetId=2600, path='/p2/phpstan/phpstan.json', kind='PACKAGE', componentId=null, component=null, assetBlobId=577941, assetBlob=AssetBlobData{assetBlobId=577941, blobRef=nexus-composer-proxy@873713b8-b626-4ad5-88f4-4c89007eb49d, blobSize=182117, contentType='application/json', checksums={sha1=aa994748be8d221e2dba828360d45cb80045b3e6, md5=f3512894185ed679b616130cf1547690, sha256=de6dac16004ef0747a6b5d858915fcb0453f8bb3ca5546ddec0ba47ee7db9408}, blobCreated=2024-09-20T06:04:04.085Z, createdBy='anonymous', createdByIp='10.9.2.119'}, lastDownloaded=2024-09-08T21:17:39.887729Z, assetSize=182117} AbstractRepositoryContent{repositoryId=4, attributes=NestedAttributesMap{parent=null, key='attributes', backing=null}, created=2024-09-08T21:17:39.887667Z, lastUpdated=2024-09-20T06:04:04.087146Z}, last_modified=2024-09-20T06:04:04.085Z}'}, assuming stale content.

I noticed that by default Maximum component age and Maximum metadate age for the repository are set to 1440 minutes (24 hours). This is probably also related to the speed of growth of the blob store size. Since the cache got corrupted in a day, the old one was not deleted and a new one was added.

Thanks.

stklcode commented 1 day ago

Thanks for the info, think I can work with that.

stklcode commented 16 hours ago

Confirmed. Looks like caching does not work at all due to this issue, so the very same file is fetched over and over again.

Think I figured out the problem, we're missing a markAsCached() call when storing blobs. Will push a PR after an additional test cycle.