Closed mark2185 closed 1 year ago
Hey, thanks for reaching out.
There's new docs for this for when v1.0.3 is released, but they aren't visible yet on the main doc site. I'll link you to them here.
tl;dr you'll need to set the STORAGE_TYPE
variable to either s3
or local
. I'm assuming you likely don't want to connect yourself to an object storage service, so you'll want to set it to local. Compose will handle creating volumes and the rest.
but I wanted to build the
release
branch
Are you sure you built that branch? Release doesn't have any code that would cause your Region is missing
error. I merged Anki imports into develop
yesterday while I tidy a few things up before the next release. If you want to test the new functionality, you'll want to use develop
.
Are you sure you built that branch?
Silly me, it was develop
all along.
tl;dr you'll need to set the STORAGE_TYPE variable to either s3 or local
Even after adding STORAGE_TYPE=local
to my .env
, I still get the same error.
Unfortunately I have close to zero experience with things that end in script
so I'm afraid all I can offer are logs:
$> docker compose --env-file .env up
WARN[0000] The "RECAPTCHA_SITE" variable is not set. Defaulting to a blank string.
WARN[0000] The "RECAPTCHA_SECRET" variable is not set. Defaulting to a blank string.
[+] Running 3/0
✔ Container scholarsome-mariadb-1 Created 0.0s
✔ Container scholarsome-redis-1 Created 0.0s
✔ Container scholarsome-scholarsome-1 Created 0.0s
Attaching to scholarsome-mariadb-1, scholarsome-redis-1, scholarsome-scholarsome-1
scholarsome-redis-1 | 1:C 04 Jul 2023 18:18:44.992 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
scholarsome-redis-1 | 1:C 04 Jul 2023 18:18:44.992 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
scholarsome-redis-1 | 1:C 04 Jul 2023 18:18:44.992 # Configuration loaded
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.993 * monotonic clock: POSIX clock_gettime
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.993 * Running mode=standalone, port=6379.
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 # Server initialized
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * Loading RDB produced by version 7.0.11
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * RDB age 9 seconds
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * RDB memory usage when created 0.82 Mb
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * Done loading RDB, keys loaded: 1, keys expired: 0.
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * DB loaded from disk: 0.000 seconds
scholarsome-redis-1 | 1:M 04 Jul 2023 18:18:44.994 * Ready to accept connections
scholarsome-mariadb-1 | 2023-07-04 18:18:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
scholarsome-mariadb-1 | 2023-07-04 18:18:45+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
scholarsome-mariadb-1 | 2023-07-04 18:18:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
scholarsome-mariadb-1 | 2023-07-04 18:18:45+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] Starting MariaDB 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 1
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Number of transaction pools: 1
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Using liburing
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Completed initialization of buffer pool
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Opened 3 undo tablespaces
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: log sequence number 187297; transaction id 256
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] Plugin 'FEEDBACK' is disabled.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] Plugin 'wsrep-provider' is disabled.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] InnoDB: Buffer pool(s) load completed at 230704 18:18:45
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] Server socket created on IP: '0.0.0.0'.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] Server socket created on IP: '::'.
scholarsome-mariadb-1 | 2023-07-04 18:18:45 0 [Note] mariadbd: ready for connections.
scholarsome-mariadb-1 | Version: '11.0.2-MariaDB-1:11.0.2+maria~ubu2204' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 | > scholarsome@1.0.3 serve:node
scholarsome-scholarsome-1 | > npm run migrate && node dist/apps/api/main.js
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 | > scholarsome@1.0.3 migrate
scholarsome-scholarsome-1 | > npx prisma migrate deploy
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 | Environment variables loaded from .env
scholarsome-scholarsome-1 | Prisma schema loaded from prisma/schema.prisma
scholarsome-scholarsome-1 | Datasource "db": MySQL database "scholarsome" at "mariadb:3306"
scholarsome-mariadb-1 | 2023-07-04 18:18:46 3 [Warning] Aborted connection 3 to db: 'scholarsome' user: 'scholarsome' host: '172.21.0.4' (Got an error reading communication packets)
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 | 10 migrations found in prisma/migrations
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 |
scholarsome-scholarsome-1 | No pending migrations to apply.
scholarsome-scholarsome-1 |
scholarsome-mariadb-1 | 2023-07-04 18:18:46 4 [Warning] Aborted connection 4 to db: 'scholarsome' user: 'scholarsome' host: '172.21.0.4' (Got an error reading communication packets)
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [NestFactory] Starting Nest application...
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] S3Module dependencies initialized +36ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] DatabaseModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] PassportModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] MailerModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] ThrottlerModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] HttpModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] ServeStaticModule dependencies initialized +1ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM LOG [InstanceLoader] UsersModule dependencies initialized +1ms
scholarsome-scholarsome-1 | [Nest] 119 - 07/04/2023, 6:18:47 PM ERROR [ExceptionHandler] Region is missing
scholarsome-scholarsome-1 | Error: Region is missing
scholarsome-scholarsome-1 | at resolveRegionConfig (/usr/src/app/node_modules/@aws-sdk/config-resolver/dist-cjs/regionConfig/resolveRegionConfig.js:9:15)
scholarsome-scholarsome-1 | at new S3Client (/usr/src/app/node_modules/@aws-sdk/client-s3/dist-cjs/S3Client.js:24:69)
scholarsome-scholarsome-1 | at new S3 (/usr/src/app/node_modules/@aws-sdk/client-s3/dist-cjs/S3.js:194:1)
scholarsome-scholarsome-1 | at createS3Connection (/usr/src/app/node_modules/nestjs-s3/dist/s3.utils.js:16:12)
scholarsome-scholarsome-1 | at InstanceWrapper.useFactory [as metatype] (/usr/src/app/node_modules/nestjs-s3/dist/s3.core-module.js:33:58)
scholarsome-scholarsome-1 | at Injector.instantiateClass (/usr/src/app/node_modules/@nestjs/core/injector/injector.js:343:55)
scholarsome-scholarsome-1 | at callback (/usr/src/app/node_modules/@nestjs/core/injector/injector.js:53:45)
scholarsome-scholarsome-1 | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
scholarsome-scholarsome-1 | at async Injector.resolveConstructorParams (/usr/src/app/node_modules/@nestjs/core/injector/injector.js:132:24)
scholarsome-scholarsome-1 | at async Injector.loadInstance (/usr/src/app/node_modules/@nestjs/core/injector/injector.js:57:13)
And yes, that is including b100584b48fd4dac7d98853aedd2c16beca0fc74.
Could you share your environment file? Redact your token info.
Sure!
I took the .env.compose.example
and edited it, and even diffed it with the latest commit to see if I'm missing something:
# This example environment file is for installations that use Docker Compose
NODE_ENV=production
# Password to provide for internal databases
DATABASE_PASSWORD=databasepassword123
# Random secret used to validate tokens
JWT_SECRET=thisisarandomjwtsecretstring
# Port to expose Scholarsome on
HTTP_PORT=8090
# Data storage configuration
# If local, file storage will be managed by Docker Compose
STORAGE_TYPE=local
# Required if storage type is s3
S3_STORAGE_ENDPOINT=
S3_STORAGE_ACCESS_KEY=
S3_STORAGE_SECRET_KEY=
S3_STORAGE_REGION=
S3_STORAGE_BUCKET=
# ---
# Everything past this line is optional
# ---
# Self-explanatory SMTP fields
# Necessary for emails to be sent - otherwise, users are verified by default
SMTP_HOST=
SMTP_PORT=
SMTP_USERNAME=
SMTP_PASSWORD=
# Domain to use in emails - do not include http
HOST=
# SSL key and cert encoded as base64
SSL_KEY_BASE64=
SSL_CERT_BASE64=
# Recaptcha site key and secret key
SCHOLARSOME_RECAPTCHA_SITE=
SCHOLARSOME_RECAPTCHA_SECRET=
# Additional scripts to be placed on the head of every page encoded in base64
SCHOLARSOME_HEAD_SCRIPTS_BASE64=
Pull from develop and give it another try. Tested it locally and should work now. The fix I pushed was added to the file but not implemented to the main module initialization of the API.
I pulled, built the scholarsome
image the same way as before, but the error persists - Region
is missing.
Compose is likely using a cached version of the image, even though you’ve regenerated the image.
Bring down the instance with docker compose down -v
. This will remove all volumes attached. Restart Compose with docker compose --env-file .env --force-recreate up
to forcefully recreate all of the containers.
Brought it down, brought it back up (with --force-recreate
), database migrations applied and other thingies, but the end result is the same, Region is missing
.
Should genuinely be fixed now, hah. Long story short, the module I was using for the S3 connection seemed like it was trying to connect regardless of whether I actually wanted it to or not. I migrated to the native one from AWS and that rectified the issues you were getting.
Yep, that did it!
I'm guessing docker compose down -v
cleaned the volumes which in turn dropped the study set I had already created, oh well. :smiling_face_with_tear:
Two other issues popped up:
v1.0.2
release tag, while the current is v1.0.3
Create
returns a 404
, the console says it's trying to send a POST
to http://localhost:8090/api/sets
Anki
deck says that the note types in it are currently not compatible with Scholarsome (screenshot of the deck)
But I propose we close this issue since my original issue was solved, these are other, non-related, issues.
I’ll just answer here, as these all are an easy fix or not an issue.
- the orange button by the current version states 'New version is available', but it's leading to the
v1.0.2
release tag, while the current isv1.0.3
This will be fixed once the new release is pushed - you’re using an unreleased version, so it just detects that the most recent release differs from your local version number.
- trying to create a new study set, i.e. filling out the title, description, editing the first card and then pressing
Create
returns a404
, the console says it's trying to send aPOST
tohttp://localhost:8090/api/sets
- trying to import the simplest
Anki
deck says that the note types in it are currently not compatible with Scholarsome (screenshot of the deck)
You’ll need to sign out and reregister your account. When you recreated with the -v
flag, (thought you were testing locally - sorry about that!) it wiped the database, including local accounts. Currently, your browser still has cookies relating to that old account, so it makes it seem like you’re logged in when you’re not. API requests are failing because the user relating back to those cookies no longer exists.
I’ll likely be pushing out v1.0.3 tonight. Let me know if you get any issues with recreating your account or any others that crop up today.
it just detects that the most recent release differs from your local version number.
Well that seems obvious in hindsight.
thought you were testing locally - sorry about that!
No worries, I'm okay with issues such as these when fiddling and hosting things on my own, thanks for the debugging and for the quick response! Looking forward to the future of the project! :)
Deleting cookies and reregistering helped with creating study decks, but the anki
import one still fails with the same error, this is from the logs:
scholarsome-scholarsome-1 | SyntaxError: Unexpected token ( in JSON at position 0
scholarsome-scholarsome-1 | at JSON.parse (<anonymous>)
scholarsome-scholarsome-1 | at SetsService.<anonymous> (/usr/src/app/dist/apps/api/main.js:2358:57)
scholarsome-scholarsome-1 | at Generator.next (<anonymous>)
scholarsome-scholarsome-1 | at /usr/src/app/node_modules/tslib/tslib.js:118:75
scholarsome-scholarsome-1 | at new Promise (<anonymous>)
scholarsome-scholarsome-1 | at Object.__awaiter (/usr/src/app/node_modules/tslib/tslib.js:114:16)
scholarsome-scholarsome-1 | at SetsService.decodeAnkiApkg (/usr/src/app/dist/apps/api/main.js:2327:24)
scholarsome-scholarsome-1 | at SetsController.<anonymous> (/usr/src/app/dist/apps/api/main.js:1874:52)
scholarsome-scholarsome-1 | at Generator.next (<anonymous>)
scholarsome-scholarsome-1 | at fulfilled (/usr/src/app/node_modules/tslib/tslib.js:115:62)
Hmm, I’ll take a look when I’m home from work. Could you share your .apkg file?
Try a set from https://ankiweb.net/shared/decks/ to see if any of those work.
.apkg files are zips that have a JSON file contained within that houses key value pairs pertaining to the media of the set. It’s possible that your JSON file is formatted differently because your set has no media.
Hmm, I’ll take a look when I’m home from work. Could you share your .apkg file?
Of course, the deck consists only of 1 card, and as you already surmised, without any media. (I had to rename it to .zip
because github
doesn't support .apkg
)
I tried downloading this one and it fails to load (probably because of audio
files (?), but there are no unexpected parsing errors in the docker
logs.
consists only of 1 card, and as you already surmised, without any media
Took a look at it. It’s good that you caught this. My suspicion is that Anki doesn’t save the meant-to-be JSON file as JSON when there is no media present in the set. I’ll add something later and it should rectify it.
I tried downloading this one and it fails to load (probably because of
audio
files (?), but there are no unexpected parsing errors in thedocker
logs.
That’s functioning as intended. For right now, the Anki imports will only support sets that have a simple front and back.
If you look at the preview for the set, you can see it has many fields:
There’s no way for me to determine (at the moment) which side the field corresponds to. In a future release, either I’ll figure out the way Anki knows which side it belongs on, or I’ll just prompt the user when importing a set which fields they want on the term and definition side.
I'd consider this solved, thanks once again for the help! :)
Hey!
Loving the project so far, and I wanted to give the new Anki import a shot before it is released since I've got some decks there, but I'm unable to start the
compose
once I build thedocker image
manually.So far I've only launched via
docker compose --env-file .env up
and that works, but I wanted to build therelease
branch and use that for thescholarsome
service. I just rundocker build -t scholarsome .
and replace theimage: hwgilbert16/scholarsome
withimage: scholarsome
, but I get an error, and it starts restarting (and looping):What am I doing wrong?
P.S. I'd ask on some forum/discord, but couldn't find any info if there are such things.